Data processing method and device based on distributed file system, equipment and medium

Abstract

The invention provides a data processing method and device based on a distributed file system, equipment and a medium, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent driving and the like. The method comprises the following steps: receiving a file operation request sent by at least one client, and determining request quota information of a directory space required by the file operation request; the at least one client comprises at least one of a user mode client and a kernel mode client; obtaining directory quota information of a target directory corresponding to the file operation request; the directory quota information is set based on a system call function and a quota operation code; the file operation request is controlled on the basis of the request quota information, the directory quota information of the target directory and the rated quota information of each related directory of the directory tree where the target directory is located, so that a quota scene in which a user-mode client and a kernel-mode client are mixed is considered, and the application range of a quota function is widened.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method, device, equipment and medium based on a distributed file system. Background Art

[0002] Ceph is a distributed storage system that can provide object, file, and block storage services at the same time. Cephfs is a distributed file system service provided by Ceph, which can provide read and write services to multiple clients at the same time. Cephfs supports both user-mode clients and kernel-mode clients. When using file system services, users will mix and match different types of clients based on performance and ease of use.

[0003] When multiple users use a shared distributed file system service, quotas are usually used to limit the usage of each user to prevent one user from using all the system resources. However, in the related art, Cephfs only supports quotas for user-mode clients, and cannot perform quota control for kernel-mode clients; and in scenarios where user-mode clients and kernel-mode clients are mixed, the quota function cannot be used normally, which has certain application limitations. Summary of the invention

[0004] The present application provides a data processing method, device, equipment and medium based on a distributed file system to solve at least one technical problem in the prior art.

[0005] On the one hand, the present application provides a data processing method based on a distributed file system, which is applied to a metadata server, including:

[0006] Receive a file operation request sent by at least one client, and determine request quota information of a directory space required by the file operation request; the at least one client includes at least one of a user state client and a kernel state client;

[0007] Obtaining directory quota information of a target directory corresponding to the file operation request; the directory quota information is set based on a system call function and a quota operation code;

[0008] The file operation request is controlled based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located.

[0009] On the other hand, a data processing device based on a distributed file system is provided, which is applied to a metadata server, and the device includes:

[0010] A receiving module, configured to receive a file operation request sent by at least one client, and determine request quota information of a directory space required for the file operation request; the at least one client includes at least one of a user state client and a kernel state client;

[0011] An acquisition module, used to acquire directory quota information of a target directory corresponding to the file operation request; the directory quota information is set based on a system call function and a quota operation code;

[0012] The control module is used to control the file operation request based on the request quota information, the directory quota information of the target directory and the rated quota information of each related directory in the directory tree where the target directory is located.

[0013] On the other hand, an electronic device is also provided, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement any of the above-mentioned data processing methods based on a distributed file system.

[0014] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement any of the above-mentioned data processing methods based on a distributed file system.

[0015] On the other hand, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes any of the above-mentioned data processing methods based on the distributed file system.

[0016] The present application provides a data processing method, device, equipment and medium based on a distributed file system, which has the following technical effects:

[0017] The embodiment of the present application determines the request quota information of the directory space required by the file operation request by receiving a file operation request sent by at least one client; the at least one client includes at least one of a user-mode client and a kernel-mode client; obtains the directory quota information of the target directory corresponding to the file operation request; the directory quota information is set based on the system call function and the quota operation code; and controls the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located. In this way, the global quota management based on the metadata service takes into account the mixed use of user-mode clients and kernel-mode clients, and improves the application scope of the quota function and the stability of system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a quota management schematic diagram in a hybrid client cluster in the related art;

[0020] Figure 2 It is a flowchart of a data processing method based on a distributed file system provided in an embodiment of the present application;

[0021] Figure 3a This is a schematic diagram of a quota setting process provided by an embodiment of the present application;

[0022] Figure 3b This is a schematic diagram of quota master-slave replica settings provided by an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a quota control process provided by an embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of implementing quota control logic based on a tree-shaped local control algorithm provided in an embodiment of the present application;

[0025] Figure 6 It is a flowchart of a quota control process provided by an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of IO splitting of a distributed file system provided in an embodiment of the present application;

[0027] Figure 8is a schematic diagram of another quota control process provided by an embodiment of the present application;

[0028] Fig. 9 It is a structural block diagram of a data processing device based on a distributed file system provided in an embodiment of the present application;

[0029] Fig.10 It is a hardware structure diagram of an electronic device provided by the present application for implementing the method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0031] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0032] In order to facilitate understanding of the above technical solutions in the embodiments of the present application, a brief introduction is given to the technologies or key terms involved in the embodiments of the present application:

[0033] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0034] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.

[0035] Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0036] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices.

[0037] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0038] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0039] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0040] Ceph: An open source distributed storage system that can provide object, file, and block storage services at the same time.

[0041] OSD: Ceph's data storage service, used to manage the reading and writing of disk data.

[0042] MDS: Cephfs metadata service, mainly provides metadata operations for the file system and the directory tree structure of the file system.

[0043] Rados: Ceph's storage cluster cornerstone, which is composed of OSDs to provide an object-based storage system.

[0044] Cephfs: Ceph provides a distributed file system service that can provide read and write services to multiple clients at the same time.

[0045] In the related art, when multiple users use a shared distributed file system service, quotas are usually used to limit the usage of each user to prevent one user from using all the resources of the system. However, Cephfs only supports quotas for fuse user-mode clients, and quota control cannot be performed for kernel-mode clients; and in scenarios where user-mode clients and kernel-mode clients are mixed, the quota function cannot be used normally, and there are certain application limitations. In addition, the kernel-mode client does not support quotas. In scenarios where user-mode and kernel-mode clients are mixed, the quota function cannot be used normally, which reduces the user experience of Cephfs. The kernel-mode client code is integrated into the Linux kernel, so the development and code merging cycle for adding or modifying new functions of the kernel-mode client is relatively long, so it is relatively difficult to modify the kernel-mode client to adapt to quota control, which brings certain quota control difficulties.

[0046] For example, Figure 1 This is a schematic diagram of quota management in a hybrid client cluster in the related art, such as Figure 1 As shown in the figure, in a mixed cluster that includes both fuse user-mode clients and kernel-mode clients, for fuse user-mode clients, the setfattr extended attribute command is generally used to set quotas for a specified directory, such as how many files and bytes this directory can contain at most. After receiving the quota setting command, the Cephfs server will broadcast the quota information to the clients that use this directory. After receiving the user request, the Ceph-fuse client will perform quota judgment processing to determine whether the number of files and bytes exceeds the quota. For kernel-mode clients, quota broadcasts cannot be transmitted to kernel-mode clients, causing the quota function to fail for kernel-mode clients.

[0047] In view of this, the present application provides a data processing method, device, equipment and medium based on a distributed file system, in which the method determines the request quota information of the directory space required for the file operation request by receiving a file operation request sent by at least one client; the at least one client includes at least one of a user-mode client and a kernel-mode client; the directory quota information of the target directory corresponding to the file operation request is obtained; the directory quota information is set based on a system call function and a quota operation code; the file operation request is controlled based on the request quota information, the directory quota information of the target directory and the rated quota information of each related directory in the directory tree where the target directory is located. In this way, the global quota management based on the metadata service takes into account the mixed use scenario of the user-mode client and the kernel-mode client, and improves the application scope of the quota function and the stability of the system performance.

[0048] The embodiments of the present application can be applied to various scenarios, such as but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and the like.

[0049] It should be noted that this application involves user information and other related data. When the following embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0050] Figure 2 This is a flowchart of a data processing method based on a distributed file system provided in an embodiment of the present application. This application provides method operation steps as described in the embodiment or flowchart, but based on conventional or non-creative labor, more or fewer operation steps may be included. The order of steps listed in the embodiment is only one way of executing the steps among many, and does not represent the only execution order. Figure 2 As shown, the method is applied to a metadata server and may include:

[0051] S201: Receive a file operation request sent by at least one client, and determine request quota information of a directory space required by the file operation request; the at least one client includes at least one of a user state client and a kernel state client.

[0052] Among them, the distributed file system (Cephfs) can include a metadata server MDS and an object storage device OSD. Among them, MDS is used to manage the metadata in the distributed file system, and users' operations on metadata must go through MDS; OSD is the component that actually stores files, consisting of a disk and a corresponding daemon process. User requests to the distributed file system will ultimately be operated and solidified by OSD.

[0053] The client is mounted on the distributed file system (Cephfs) and can send file operation requests to the metadata server MDS therein. The obtained file operation request is parsed to obtain the request quota information of the directory space required for the file operation request. The file operation request is used to request operations such as viewing, modifying, adding, and deleting stored files. The request quota information may include the number of requested files, the number of requested bytes, etc. The client here can be at least one of a user-mode client and a kernel-mode client, for example, it can be any one of a user-mode client and a kernel-mode client, or it can include both user-mode clients and kernel-mode clients.

[0054] S203: Obtain directory quota information of the target directory corresponding to the file operation request; the directory quota information is set based on the system call function and the quota operation code.

[0055] In an optional implementation manner, before obtaining directory quota information of the target directory corresponding to the file operation request, the method further includes:

[0056] Obtaining a directory quota request, the directory quota request is generated based on directory quota information input for a directory to be set in a command prompt, the command prompt is displayed based on a system call function and a quota operation code; the directory to be set is any directory in each directory tree distributed in a metadata server;

[0057] Based on the target quota information contained in the directory quota request, the quota information in the directory to be set is adjusted;

[0058] The directory quota information includes at least one of directory file quantity quota information and directory byte quantity quota information.

[0059] Optionally, based on the system call function and the quota operation code, a command prompt (CMD) is displayed; the directory quota information input for the directory to be set in the command prompt is obtained, and a directory quota request is generated; the directory quota request is used to assign the input directory quota information to the quota attribute of the directory to be set; the directory to be set is any directory in the directory tree distributed in the metadata server. Then, the directory quota request is sent to the metadata server, and the metadata server adjusts the quota information in the directory to be set based on the target quota information included in the directory quota request. For example, at least one of the directory file quantity quota information and the directory byte quantity quota information in the quota information can be adjusted, such as adding, increasing, decreasing, etc.

[0060] In the related art, although the Fuse user-mode client supports the management of general quotas, the quota is set using the setfattr command that can be used by all users to modify the extended attribute. This can cause users to increase their quotas by modifying the extended attribute when the quota is insufficient, making the quota management confusing.

[0061] In this application, Figure 3a The following is a schematic diagram showing the process of quota setting: Figure 3a As shown in the figure, the quota setting here can only be operated by the system administrator. The administrator sets the quota for the directory to be set (for example, the / data / a directory) through the ioctl system call function and the quota opcode of CEPH_IOC_SET_QUOTA. This achieves the isolation between quota setting and quota use, so that users cannot modify the quota privately, ensuring the controllability of quota setting.

[0062] In an optional implementation, the metadata server includes a first sub-server where the master copy of the directory data is located, and a second sub-server where the slave copy of the directory data is located. The above request to obtain the directory quota includes:

[0063] If the second sub-server obtains the directory quota request, it forwards the directory quota request to the first sub-server.

[0064] Accordingly, based on the target quota information included in the directory quota request, adjusting the quota information in the directory to be set includes:

[0065] Adjusting, by the first sub-server, the attribute value of the quota attribute in the directory to be set in the first sub-server based on the target quota information included in the directory quota request;

[0066] The attribute value of the adjusted quota attribute in the first sub-server is synchronized to the corresponding directory in the second sub-server.

[0067] The metadata server here may include multiple sub-servers, that is, the metadata server is an MDS service cluster.

[0068] Exemplarily, the directory tree of the entire file system of the distributed file system is distributed in the entire MDS service cluster. The MDS service cluster is composed of multiple MDSs. Each MDS can manage a part of the distributed file system directory tree, and can dynamically adjust which part of the directory tree each MDS manages according to the directory load in the directory tree. The directory in the distributed file system can be stored in all MDSs, one of which (i.e., the first sub-server) is the master copy of the directory data, and the other MDSs (i.e., the second sub-servers) are slave copies. Exemplarily, the number of first sub-servers can be one, and the number of second sub-servers can be one or more. The master copy is used to control the modification of directory data, such as creating a sub-directory or a new file under the directory, so the quota setting request for the directory is sent to the master copy for processing, and then the master copy will synchronize the updated quota information to the slave copy.

[0069] Specifically, Figure 3b The following is a schematic diagram of quota master-slave replica settings: Figure 3b As shown, the target quota information may include how many files and bytes are allowed under the directory, that is, the two attributes of the number of files and the number of bytes. The master-slave replica quota setting process may include:

[0070] (1) Set the directory quota request to be sent to MDS_0 (i.e., the first sub-server) where the master copy is located. If the request is sent to MDS_1 (i.e., the second sub-server) where the slave copy is located, this MDS_1 will forward the directory quota request to MDS_0, the master copy where the directory is located.

[0071] (2) Modify the quota information of the primary copy directory, for example, set the quota information of the / data / a directory, that is, adjust the attribute value of the quota attribute in the directory to be set in the first sub-server, for example, adjust it to the number of files m and the number of bytes n.

[0072] (3) Synchronize the adjusted quota information in the master copy directory to the slave copy to ensure the consistency of the master and slave copy quota information, that is, synchronize the attribute values ​​of the adjusted quota attributes to the corresponding directories in the second sub-server, for example, synchronize the attribute values ​​in the / data / a directory in MDS_0 to the attribute values ​​in the / data / a directory in MDS_1.

[0073] In the above embodiment, the quota is controlled by the sub-server corresponding to the master copy in the metadata server, and the quota is further uniformly controlled by the first sub-server where the master copy is located, thereby avoiding confusion in quota control and improving the speed and accuracy of quota control. At the same time, by synchronizing the quota information modified on the first sub-server to the second sub-servers where the other slave copies are located, the consistency of the quota information stored in each sub-service in the metadata server is ensured, thereby improving the reliability and stability of quota control.

[0074] S205: Control the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located.

[0075] The related directory may include at least one parent directory in the directory tree where the target directory is located. The rated quota information refers to the maximum quota information of each related directory, such as the maximum file number quota, the maximum byte number quota, etc. The rated quota information can also be set through the ioctl system call function and the quota operation code such as CEPH_IOC_SET_QUOTA.

[0076] Optionally, quota control is centralized on the MDS metadata server and controls requests for modifying the number of files that affect the quota, such as mkdir (create directory), openc (create file), rename (rename operations such as mv), etc. It can also control input operation requests that affect the byte size of the file.

[0077] In an optional implementation, the directory quota information includes directory file quantity quota information, and the rated quota information includes a rated file quantity quota; and controlling the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located includes:

[0078] Identify the operation type of the file operation request to obtain a first type of operation request; the first type of operation request is any operation request in the file operation request for indicating an increase in the number of files;

[0079] Based on the request quota information, the directory quota information of the target directory and the rated file quantity quota of each related directory in the directory tree where the target directory is located, quota control at the file quantity level is performed on the first type of operation request.

[0080] Specifically, the directory quota information mainly includes the directory file quantity quota information and the directory byte quantity quota information. Here we first introduce the control of the directory file quantity quota information. The subdirectory in the directory tree can be regarded as a special file. For the quota control of the number of files, it is necessary to identify the user request operation to increase the number of files, that is, the first type of operation request. The first type of operation request is any operation request in the file operation request used to indicate the increase of the number of files. There are mainly the following operation requests: openc (create a new file); mknod (create a device file, such as a character device file or a block device file), mkdir (create a subdirectory), symlink (create a symbolic link), link (create a hard link), rename (rename operation, mainly mv commands), etc. The above operation requests will increase the number of files, so it is necessary to perform quota control at the file quantity level, that is, based on the request quota information, the directory quota information of the target directory, and the rated file quantity quota of each relevant directory in the directory tree where the target directory is located, the first type of operation request is subject to quota control at the file quantity level.

[0081] In an optional embodiment, the metadata server may include an operation request distribution module, a request processing module and a quota control module. The operation request distribution module is used to obtain a file operation request sent by at least one client, and forward the file operation request to the request processing module. The request processing module is used to identify the operation type of the forwarded file operation request and obtain a target class operation request. The quota control module is at least used to perform quota control on the file quantity level for the first class operation request, and feed back the quota control result to the request processing module, so that the request processing module performs corresponding operations based on the quota control result; the target class operation request includes at least one of the first class operation request for indicating an increase in the number of files and the second class operation request for performing a write operation.

[0082] Specifically, Figure 4 As shown, the metadata server includes an operation request distribution module, a request processing module and a quota control module. Among them, the request distribution module is used to receive client requests and forward the requests to the request processing module. The request processing module (i.e., the Server module) is used to identify and process client requests, such as processing the first type of operation requests such as openc, mknod, and mkdir. The quota control module is at least used to perform quota control on newly added directories. In each request processing function that increases the number of directory files, the quota control module can be used to check whether the number of files in the currently operated directory exceeds the quota.

[0083] In an optional implementation, based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located, performing quota control on the first type of operation request at the file quantity level includes:

[0084] Make the target directory the current directory;

[0085] When the current target is determined to be a non-root directory or base directory, obtain the number of recursive subtree files in the current directory;

[0086] If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is greater than the rated file number quota of the current directory, the result that the first type of operation request exceeds the quota is returned;

[0087] If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is less than or equal to the rated file number quota of the current directory, the parent directory of the current directory is used as the current directory, and the step of obtaining the number of files in the recursive subtree of the current directory is returned until the current target is the root directory or the base directory.

[0088] Alternatively, if Figure 5 and Figure 6 As shown, since the directory exists in the directory tree of the distributed file system, there is a directory subtree and its parent directory for the directory. The tree-shaped local control logic is used to control the downward and upward tracing of these two parts, as follows:

[0089] Check below: Quota control of the number of files in a directory subtree. Each directory is configured with a recursive statistical attribute rstat, which contains all recursive files and subdirectories under this directory. Therefore, the control of the directory subtree only needs to obtain the number of files in the recursive subtree of the directory through this recursive statistical attribute.

[0090] Trace back: Start from the parent directory and trace back to the root directory or base directory (each MDS has a base virtual root directory), and check whether the subdirectories under each parent directory meet the file quota requirements.

[0091] During the quota control process, if the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is greater than the rated file number quota of the current directory, the result that the first type of operation request exceeds the quota is returned, such as replying to the client Nospace (i.e., indicating that there is not enough directory space). If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is less than or equal to the rated file number quota of the current directory, the parent directory of the current directory is used as the current directory, and the step of obtaining the number of files in the recursive subtree of the current directory is continued until the current target is the root directory or the base directory, and then the file number quota control of the directory tree can be ended, and the operation corresponding to the first operation request is executed.

[0092] The above embodiment implements the control of file number quota through tree-shaped local control logic, reduces the amount of calculation for traversing the entire directory tree, and improves the quota-based data processing efficiency and system performance.

[0093] Normally, the byte quota control of a directory mainly controls the amount of data that can be written to this directory. Since Cephfs uses a design that separates the MDS-based metadata stream and the OSD-based data stream, the data read and write IO does not directly access the MDS metadata server. After the file is written through data IO, the MDS will be informed through cap (a kind of permission control in cephfs) that the file size has changed, such as Figure 7 shown.

[0094] In order to achieve the control of byte quota, the quota of directory bytes is delayed and controlled at two times, so that the byte quota is controlled within a small error, such as Figure 8 The two opportunities are as follows:

[0095] Opportunity 1: The control of the directory file quantity quota is triggered at the same time, that is, the byte quantity quota is also judged and controlled when the number of files changes.

[0096] In an optional implementation, the directory quota information further includes directory byte quantity quota information, and the rated quota information includes a rated byte quantity quota; and controlling the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located further includes:

[0097] In response to a triggering opportunity for executing quota control on a first type of operation request, quota control is performed on the first type of operation request at the byte quantity level based on the request quota information, the directory quota information of the target directory, and the rated byte quantity quotas of each relevant directory in the directory tree where the target directory is located.

[0098] In an optional implementation, the quota control at the byte quantity level here may also adopt the control logic of tree-shaped local control. In this case, based on the request quota information, the directory quota information of the target directory, and the rated byte quantity quota of each related directory in the directory tree where the target directory is located, the quota control at the byte quantity level for the first type of operation request includes:

[0099] Make the target directory the current directory;

[0100] When the current target is determined to be a non-root directory or base directory, the number of bytes of the recursive subtree of the current directory is obtained;

[0101] If the sum of the number of bytes in the recursive subtree and the number of bytes requested corresponding to the first type of operation request is greater than the rated byte quota of the current directory, the result that the first type of operation request exceeds the quota is returned;

[0102] If the sum of the number of bytes in the recursive subtree and the number of bytes requested corresponding to the first type of operation request is less than or equal to the rated byte quota of the current directory, the parent directory of the current directory is used as the current directory, and the step of obtaining the number of recursive byte files in the current directory is returned until the current target is the root directory or the base directory.

[0103] Optionally, during the quota control process, if the sum of the number of bytes of the recursive subtree and the number of requested files corresponding to the first type of operation request is greater than the rated byte quota of the current directory, the result that the first type of operation request exceeds the byte quota is returned, such as replying to the client Nospace (i.e., indicating that there is not enough directory space). If the sum of the number of bytes of the recursive subtree and the number of requested bytes corresponding to the first type of operation request is less than or equal to the rated byte quota of the current directory, the parent directory of the current directory is used as the current directory, and the step of obtaining the number of bytes of the recursive subtree of the current directory is continued until the current target is the root directory or the base directory, and then the quota control of the directory tree can be ended, and the operation corresponding to the first operation request is executed.

[0104] Opportunity 2: Control the byte quota when the file is opened. Usually, the trigger for opening the file again is shortly after the write operation.

[0105] In an optional implementation, the directory quota information further includes directory byte quantity quota information, and the rated quota information includes a rated byte quantity quota; and controlling the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located further includes:

[0106] Determine any second-category operation request for performing a write operation on a target file in the file operation request;

[0107] In response to the triggering opportunity of opening the target file again, based on the request quota information, the directory quota information of the target directory and the rated byte quota of each related directory in the directory tree where the target directory is located, the second type of operation request is subjected to delayed quota control at the byte quantity level.

[0108] Optionally, since data read and write IO and metadata read and write IO are separate, the byte quantity quota is not controlled during write IO, but is controlled in subsequent operations. For example, after writing a data block, the target file corresponding to the database will be closed, and the target file will be opened again for writing soon. The quota control can be delayed until the next triggering time of the operation of opening the target file, so as to implement delayed quota control at the byte quantity level for the second type of operation request for executing write.

[0109] The above embodiment performs delay quota control at two opportunities, provides multiple delay control mechanisms, reduces quota control errors, and improves the accuracy of quota control based on the byte quantity level.

[0110] In an optional embodiment, the method further comprises:

[0111] When the quota control result indicates that the rated quota information is satisfied, the operation corresponding to the file operation request is executed.

[0112] Optionally, when the quota control result based on the file quantity level and the quota control result based on the byte quantity level indicate that the rated quota information is satisfied, a processing operation corresponding to the file operation request is performed. The corresponding processing method may be different according to the difference of each file operation request, for example, an openc request is to create a file, and an mkdir request is to create a folder; according to the difference of the object to be operated, the corresponding metadata is created or modified.

[0113] In the above embodiment, by receiving a file operation request sent by at least one client, the request quota information of the directory space required by the file operation request is determined; the at least one client includes at least one of a user-mode client and a kernel-mode client; the directory quota information of the target directory corresponding to the file operation request is obtained; the directory quota information is set based on the system call function and the quota operation code; the file operation request is controlled based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located. In this way, the global quota management based on the metadata service takes into account the mixed use of user-mode clients and kernel-mode clients, and improves the application scope of the quota function and the stability of system performance.

[0114] In addition, by providing unified quota control in the metadata MDS cluster, quota control is implemented on the server side through tree-based local control and delayed byte control, so that the quota of the distributed file system can be used in both the fuse user-mode client and the kernel-mode client, improving the user experience of the distributed file system.

[0115] The following is an embodiment of the device of the present application, which can be used to perform the steps in the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0116] Fig. 9 is a structural block diagram of a data processing device based on a distributed file system provided in an embodiment of the present application. The data processing device based on a distributed file system has the functions of implementing the above method examples, and the functions can be implemented by hardware or by hardware executing corresponding software. Fig. 9 As shown, a data processing device based on a distributed file system is applied to a metadata server, and the device may include:

[0117] The receiving module 910 is used to receive a file operation request sent by at least one client, and determine the request quota information of the directory space required by the file operation request; the at least one client includes at least one of a user state client and a kernel state client;

[0118] The acquisition module 920 is used to acquire directory quota information of the target directory corresponding to the file operation request; the directory quota information is set based on the system call function and the quota operation code;

[0119] The control module 930 is used to control the file operation request based on the request quota information, the directory quota information of the target directory and the rated quota information of each related directory in the directory tree where the target directory is located.

[0120] In an optional embodiment, the device further comprises:

[0121] A quota request acquisition module is used to acquire a directory quota request, the directory quota request is generated based on the directory quota information input for the directory to be set in the command prompt, the command prompt is displayed based on the system call function and the quota operation code; the directory to be set is any directory in each directory tree distributed in the metadata server;

[0122] A quota adjustment module, used for adjusting the quota information in the directory to be set based on the target quota information contained in the directory quota request;

[0123] The directory quota information includes at least one of directory file quantity quota information and directory byte quantity quota information.

[0124] In an optional implementation, the metadata server includes a first sub-server where the master copy of the directory data is located, and a second sub-server where the slave copy of the directory data is located; the acquisition module is further used to:

[0125] If the second sub-server obtains the directory quota request, forwarding the directory quota request to the first sub-server;

[0126] Accordingly, the quota adjustment module is also used to:

[0127] Adjusting, by the first sub-server, the attribute value of the quota attribute in the directory to be set in the first sub-server based on the target quota information included in the directory quota request;

[0128] The attribute value of the adjusted quota attribute in the first sub-server is synchronized to the corresponding directory in the second sub-server.

[0129] In an optional implementation manner, the directory quota information includes directory file quantity quota information, and the rated quota information includes a rated file quantity quota; and the control module is further configured to:

[0130] Identify the operation type of the file operation request to obtain a first type of operation request; the first type of operation request is any operation request in the file operation request for indicating an increase in the number of files;

[0131] Based on the request quota information, the directory quota information of the target directory and the rated file quantity quotas of each relevant directory in the directory tree where the target directory is located, quota control at the file quantity level is performed on the first type of operation request; the target type operation request is an operation in at least one type of operation request that is used to indicate an increase in the number of files.

[0132] In an optional implementation manner, the control module is specifically used to:

[0133] Make the target directory the current directory;

[0134] When the current target is determined to be a non-root directory or base directory, obtain the number of recursive subtree files in the current directory;

[0135] If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is greater than the rated file number quota of the current directory, the result that the first type of operation request exceeds the quota is returned;

[0136] If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is less than or equal to the rated file number quota of the current directory, the parent directory of the current directory is used as the current directory, and the step of obtaining the number of files in the recursive subtree of the current directory is returned until the current target is the root directory or the base directory.

[0137] In an optional implementation manner, the directory quota information further includes directory byte quantity quota information, and the rated quota information includes the rated byte quantity quota; and the control module is further configured to:

[0138] In response to a triggering opportunity for executing quota control on a first type of operation request, quota control is performed on the first type of operation request at the byte quantity level based on the request quota information, the directory quota information of the target directory, and the rated byte quantity quotas of each relevant directory in the directory tree where the target directory is located.

[0139] In an optional implementation manner, the control module is further specifically configured to:

[0140] Make the target directory the current directory;

[0141] When the current target is determined to be a non-root directory or base directory, the number of bytes of the recursive subtree of the current directory is obtained;

[0142] If the sum of the number of bytes in the recursive subtree and the number of bytes requested corresponding to the first type of operation request is greater than the rated byte quota of the current directory, the result that the first type of operation request exceeds the quota is returned;

[0143] If the sum of the number of bytes in the recursive subtree and the number of bytes requested corresponding to the first type of operation request is less than or equal to the rated byte quota of the current directory, the parent directory of the current directory is used as the current directory, and the step of obtaining the number of recursive byte files in the current directory is returned until the current target is the root directory or the base directory.

[0144] In an optional implementation manner, the directory quota information further includes directory byte quantity quota information, and the rated quota information includes the rated byte quantity quota; and the control module is further configured to:

[0145] Determine any second-category operation request for performing a write operation on a target file in the file operation request;

[0146] In response to the triggering opportunity of opening the target file again, based on the request quota information, the directory quota information of the target directory and the rated byte quota of each related directory in the directory tree where the target directory is located, the second type of operation request is subjected to delayed quota control at the byte quantity level.

[0147] In an optional embodiment, the device further comprises:

[0148] When the quota control result indicates that the rated quota information is satisfied, the operation corresponding to the file operation request is executed.

[0149] It should be noted that: when executing the data processing device based on the distributed file system provided in the above embodiment, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device based on the distributed file system provided in the above embodiment belongs to the same concept as the data processing method embodiment based on the distributed file system, and the data processing device based on the distributed file system provided in the above embodiment belongs to the same concept as the data processing method embodiment based on the distributed file system. For technical details not described in detail in the above device embodiment, please refer to the method provided in any embodiment of the present application, and will not be repeated here.

[0150] An embodiment of the present application provides an electronic device, which may include a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a method as described in any of the above method embodiments.

[0151] Further, Fig.10 1 is a schematic diagram of the hardware structure of an electronic device for implementing the method provided in the embodiment of the present application. Fig.10 The electronic device includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps of any method in the above embodiments when executing the instructions stored in the memory.

[0152] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.10It is a block diagram of an electronic device for data processing based on a distributed file system according to an exemplary embodiment. The electronic device 1000 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1010 (the processor 1010 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1030 for storing data, and one or more storage media 1020 (such as one or more mass storage devices) for storing application programs 1023 or data 1022. Among them, the memory 1030 and the storage medium 1020 can be short-term storage or permanent storage. The program stored in the storage medium 1020 may include one or more modules, and each module may include a series of instruction operations in the electronic device. Furthermore, the central processor 1010 can be configured to communicate with the storage medium 1020 and execute a series of instruction operations in the storage medium 1020 on the electronic device 1000.

[0153] The electronic device 1000 may also include one or more power supplies 1060, one or more wired or wireless network interfaces 1050, one or more input and output interfaces 1040, and / or one or more operating systems 1021, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.

[0154] The input / output interface 1040 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 1000. In one example, the input / output interface 1040 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In an exemplary embodiment, the input / output interface 1040 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0155] It can be understood by those skilled in the art that Fig.10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.10 More or fewer components as shown, or with Fig.10 Different configurations shown.

[0156] The embodiment of the present application also provides a computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, and the at least one instruction or the at least one program is loaded and executed by a processor to implement any of the above methods. For example, the storage medium includes a memory of instructions, and the above instructions can be executed by a processor of the electronic device 1000 to complete the above method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0157] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes any of the above-mentioned methods.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0160] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0161] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0162] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method based on a distributed file system, characterized in that: Applies to metadata servers, including: Receive a file operation request sent by at least one client, and determine request quota information of a directory space required by the file operation request; the at least one client includes at least one of a user state client and a kernel state client; Obtaining directory quota information of a target directory corresponding to the file operation request; the directory quota information is set based on a system call function and a quota operation code; The file operation request is controlled based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located.

2. The method according to claim 1, characterized in that Before obtaining the directory quota information of the target directory corresponding to the file operation request, the method further includes: Obtaining a directory quota request, wherein the directory quota request is generated based on directory quota information input for a directory to be set in a command prompt, wherein the command prompt is displayed based on a system call function and a quota operation code; the directory to be set is any directory in each directory tree distributed in the metadata server; Adjusting the quota information in the directory to be set based on the target quota information included in the directory quota request; The directory quota information includes at least one of directory file quantity quota information and directory byte quantity quota information.

3. The method according to claim 1, characterized in that The metadata server includes a first sub-server where a master copy of the directory data is located, and a second sub-server where a slave copy of the directory data is located; The request to obtain directory quota includes: If the second sub-server obtains the directory quota request, forwarding the directory quota request to the first sub-server; Correspondingly, adjusting the quota information in the directory to be set based on the target quota information included in the directory quota request includes: adjusting, by the first sub-server, the attribute value of the quota attribute in the directory to be set in the first sub-server based on the target quota information included in the directory quota request; The attribute value of the adjusted quota attribute in the first sub-server is synchronized to the corresponding directory in the second sub-server.

4. The method according to claim 1, characterized in that The directory quota information includes directory file quantity quota information, and the rated quota information includes a rated file quantity quota; the controlling the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located includes: Identify the operation type of the file operation request to obtain a first type of operation request; the first type of operation request is any operation request in the file operation request for indicating an increase in the number of files; Based on the request quota information, the directory quota information of the target directory and the rated file quantity quota of each related directory in the directory tree where the target directory is located, quota control at the file quantity level is performed on the first type of operation request.

5. The method according to claim 4, characterized in that The performing quota control on the file quantity level for the first type of operation request based on the request quota information, the directory quota information of the target directory, and the rated file quantity quota of each related directory in the directory tree where the target directory is located comprises: Set the target directory as the current directory; When the current target is determined to be a non-root directory or base directory, obtain the number of recursive subtree files in the current directory; If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is greater than the rated file number quota of the current directory, a result indicating that the first type of operation request exceeds the quota is returned; If the sum of the number of files in the recursive subtree and the number of requested files corresponding to the first type of operation request is less than or equal to the rated file number quota of the current directory, take the parent directory of the current directory as the current directory, and return to the step of obtaining the number of files in the recursive subtree of the current directory until the current target is the root directory or the base directory.

6. The method according to claim 4, characterized in that The directory quota information also includes directory byte quantity quota information, and the rated quota information includes a rated byte quantity quota; the controlling the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located also includes: In response to a triggering opportunity for executing quota control on the first type of operation request, quota control is performed on the first type of operation request at the byte quantity level based on the request quota information, the directory quota information of the target directory, and the rated byte quantity quota of each relevant directory in the directory tree where the target directory is located.

7. The method according to claim 6, characterized in that The performing byte quantity-level quota control on the first type of operation request based on the request quota information, the directory quota information of the target directory, and the rated byte quantity quota of each related directory in the directory tree where the target directory is located includes: Set the target directory as the current directory; When the current target is determined to be a non-root directory or base directory, the number of bytes of the recursive subtree of the current directory is obtained; If the sum of the number of bytes of the recursive subtree and the number of bytes requested corresponding to the first type of operation request is greater than the rated byte quota of the current directory, a result indicating that the first type of operation request exceeds the quota is returned; If the sum of the number of bytes of the recursive subtree and the number of requested bytes corresponding to the first type of operation request is less than or equal to the rated byte quota of the current directory, take the parent directory of the current directory as the current directory, and return to the step of obtaining the number of recursive byte files of the current directory until the current target is the root directory or the base directory.

8. The method according to any one of claims 1 to 7, characterized in that: The directory quota information also includes directory byte quantity quota information, and the rated quota information includes a rated byte quantity quota; the controlling the file operation request based on the request quota information, the directory quota information of the target directory, and the rated quota information of each related directory in the directory tree where the target directory is located also includes: Determine any second-category operation request for performing a write operation on a target file in the file operation request; In response to a triggering opportunity to open the target file again, based on the request quota information, the directory quota information of the target directory and the rated byte quota of each related directory in the directory tree where the target directory is located, delayed quota control is performed on the second type of operation request at the byte quantity level.

9. The method according to any one of claims 4 to 7, characterized in that: The method further comprises: When the quota control result indicates that the rated quota information is satisfied, the operation corresponding to the file operation request is executed.

10. A data processing device based on a distributed file system, characterized in that: Applied to a metadata server, the device comprises: A receiving module, configured to receive a file operation request sent by at least one client, and determine request quota information of a directory space required for the file operation request; the at least one client includes at least one of a user state client and a kernel state client; An acquisition module, used to acquire directory quota information of a target directory corresponding to the file operation request; the directory quota information is set based on a system call function and a quota operation code; The control module is used to control the file operation request based on the request quota information, the directory quota information of the target directory and the rated quota information of each related directory in the directory tree where the target directory is located.

11. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method based on a distributed file system as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method based on the distributed file system as described in any one of claims 1-9.

Images