A method, system, device, and media for optimizing directory quotas in a distributed file system.

Abstract

This invention belongs to the field of computer science, specifically relating to a method, system, device, and medium for optimizing directory quotas in a distributed file system. The method includes: generating different pre-allocation quotas in the distributed file system according to a predetermined method, and responding to client pre-allocation requests with different pre-allocation quotas; evaluating the operational data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota; and, in response to determining the optimal pre-allocation quota, applying the optimal pre-allocation quota to the distributed file system. Through this distributed file system directory quota optimization method, the impact of different pre-allocation quotas on the distributed file system is analyzed based on the operational data of the distributed file system under different pre-allocation quotas, and the optimal pre-allocation quota is identified and applied to the distributed file system to improve its performance.

Description

Technical Field

[0001] This invention belongs to the field of computer science, and specifically relates to a method, system, device, and medium for optimizing directory quotas in a distributed file system. Background Technology

[0002] In the storage field, to improve the overall resource utilization of a distributed file system (DFS), quotas, or resource limits, are used to manage and control file system resources. This allows users to customize resource allocation for target users, user groups, and specified directories, efficiently utilizing the system's storage space. In DFS, directory quotas are centered around directories, and settings can be used to count and limit the number of files or the directory size.

[0003] Quota statistics are stored on the Metadata Server (MDS). Each client periodically or quantitatively reports quota statistics to the MDS. The MDS data serves as the basis for other servers and clients to update their data. Each client can query the latest quota information from the MDS and cache it locally. After the system enables directory quota functionality, when the file size in a directory with quota information changes, confirming the increase, the client sends a pre-request quota to the MDS server. This pre-request includes a fixed-size quota request, such as 100MB. Once the MDS server confirms that the pre-request does not exceed the quota setting limit, it allocates the directory quota to the corresponding client.

[0004] Quota pre-request is triggered when the locally stored quota information determines that the remaining quota pre-request quantity is less than the increase in directory capacity. In cases involving file deletion or a decrease in system capacity, the quota information is automatically updated and reported to MDS periodically.

[0005] In other words, using quota pre-allocation can effectively reduce the number of interactions between the client and MDS, improving directory IO performance under quota limits. However, the current quota pre-allocation quantity uses a fixed value. When the pre-allocation quota value is set too small, for large files, it is still necessary to frequently send quota pre-allocation requests to MDS. When the pre-allocation quota value is set too large, the granularity of the system's quota limit is not fine enough, which can easily lead to a large error between the quota limit and the actual capacity.

[0006] Therefore, an effective solution is urgently needed to address the above problems. Summary of the Invention

[0007] To address the above problems, this invention proposes a distributed file system directory quota optimization method, comprising:

[0008] Different pre-allocation quotas are generated in the distributed file system according to a predetermined method, and the client's pre-allocation request is responded to with different pre-allocation quotas;

[0009] The distributed file system's operational data under different pre-allocation quotas is evaluated according to a predetermined strategy to obtain the optimal pre-allocation quota.

[0010] In response to determining the optimal pre-allocated quota, the optimal pre-allocated quota is applied to the distributed file system.

[0011] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota includes:

[0012] Different pre-request quotas are evaluated based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota.

[0013] In some embodiments of the present invention, evaluating different pre-request quotas based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota includes:

[0014] Obtain the data transfer request volume and pre-allocation request volume processed by the distributed file system under different pre-allocation quota scenarios;

[0015] The ratio of the data transmission request volume to the pre-requested quota volume is used as the score for the pre-requested quota;

[0016] By comparing the scores of different pre-application quotas, the pre-application quota with the highest score is selected as the optimal pre-application quota.

[0017] In some embodiments of the present invention, the method further includes:

[0018] The idle rate of the corresponding pre-requested quota is calculated based on the idle space of each client;

[0019] The final score of the pre-requested quota is obtained by subtracting the idle rate corresponding to the pre-requested quota from the score of the pre-requested quota.

[0020] By comparing the final scores of different pre-application quotas, the pre-application quota with the highest final score is selected as the optimal pre-application quota.

[0021] In some embodiments of the present invention, generating different pre-allocated quotas in a distributed file system according to a predetermined method includes:

[0022] Obtain the total capacity of the distributed file system, and use a predetermined proportion of the total capacity as the maximum value of the pre-requested quota;

[0023] Determine the minimum value of the pre-request quota, and generate multiple pre-request quotas between the maximum and minimum values ​​of the pre-request quota with a predetermined step size.

[0024] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota further includes:

[0025] Based on the file specifications of the data transmission request, the task volume of the corresponding pre-request quota is classified according to the data transmission request and the pre-request request. Under different categories, different pre-request quotas are evaluated based on the task volume of the data transmission request and the pre-request request to determine the optimal pre-request quota.

[0026] In some embodiments of the present invention, the method further includes:

[0027] The AI ​​model is trained based on the business operation data of the distributed file system under different pre-request quotas, and the corresponding pre-request quota is output based on the data transmission request based on the AI ​​model.

[0028] Another aspect of the present invention proposes a distributed file system directory quota optimization system, comprising:

[0029] A pre-request quota generation module is configured to generate different pre-request quotas in a distributed file system according to a predetermined method, and respond to the client's pre-request request with different pre-request quotas;

[0030] A pre-request quota evaluation module is configured to evaluate the business operation data of the distributed file system under different pre-request quotas according to a predetermined strategy in order to obtain the optimal pre-request quota.

[0031] A pre-allocation quota implementation module is configured to apply the optimal pre-allocation quota to the distributed file system in response to determining the optimal pre-allocation quota.

[0032] Another aspect of the present invention provides a computer device comprising:

[0033] At least one processor; and

[0034] A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of any of the methods described in the above embodiments.

[0035] Another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the above embodiments.

[0036] The present invention proposes a distributed file system directory quota optimization method, which analyzes the impact of different pre-allocation quotas on the distributed file system based on the operating data of the distributed file system under different pre-allocation quotas, finds the optimal pre-allocation quota, and applies the optimal pre-allocation quota to the distributed file system to improve the performance of the distributed file system. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A flowchart illustrating a distributed file system directory quota optimization method provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of a distributed file system directory quota optimization system provided in an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention;

[0041] Figure 4 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0043] This invention aims to solve the problem of exceeding the directory file size when multiple clients write files to a shared file directory in a distributed file system. In a distributed file system, multiple clients can write files to the same directory and can write simultaneously in parallel. For common distributed file systems (DFS), a metadata server (MDS) is generally used to coordinate multiple clients writing data to the file directory. When a client writes data to the file directory, it first requests a certain amount of data space from the metadata server. The metadata server then allocates a uniform quota space to each client, i.e., a pre-allocation quota method. While the pre-allocation quota method can effectively reduce the number of interactions between the client and the MDS and improve the directory's I / O performance under quota constraints, the current fixed quota pre-allocation value has limitations. When the value is set too small, for large files, frequent and repeated quota pre-allocation requests to the MDS are still required. When the value is set too large, the granularity of the quota constraint is not fine enough, which can easily lead to a large error between the quota limit and the actual capacity.

[0044] like Figure 1 As shown, to solve the above problems, this invention proposes a distributed file system directory quota optimization method, including:

[0045] Step S1: Generate different pre-request quotas in the distributed file system according to a predetermined method, and respond to the client's pre-request request with different pre-request quotas;

[0046] Step S2: Evaluate the business operation data of the distributed file system under different pre-request quotas according to the predetermined strategy to obtain the optimal pre-request quota;

[0047] Step S3: In response to determining the optimal pre-allocation quota, apply the optimal pre-allocation quota to the distributed file system.

[0048] In an embodiment of the present invention, the present invention is applied to a distributed file system. In the distributed file system, the metadata server in the distributed file system responds to each client's writing of data to the file directory by pre-allocating quotas. The pre-allocating quotas control the "rhythm" of each client's writing of data to the file directory. That is, only when the client sends a pre-allocating request for data transmission to the metadata server, and the metadata server responds to the corresponding data transmission request based on the available space remaining capacity of the directory file, and "limits" the amount of data written by the client at this time by pre-allocating quotas, can the client write data to the directory file.

[0049] Therefore, if the pre-request quota is not set properly, the client will frequently send pre-request requests to the metadata server to request the corresponding data storage space. Moreover, there is more than one client, so the frequent sending of pre-request requests will greatly increase the pressure on the metadata server.

[0050] Furthermore, to address the aforementioned issues, in step S1, multiple pre-allocation quotas of varying sizes are set on the distributed file system in a progressively increasing manner, and each pre-allocation quota is independently applied to one end of the distributed file system, enabling the metadata server to respond to pre-allocation requests from clients based on each pre-allocation quota.

[0051] In step S2, the operating data of the distributed file system under different pre-allocation quotas is obtained. This operating data includes: the number of pre-allocation requests received by the distributed file system, the number of data transfer requests, the current directory capacity, the directory offset capacity at different times (i.e., the storage space requested by each client), the error value between the quota limit and the actual final write capacity, and the number of interactions with the MDS server. The operating data of the distributed file system is analyzed to determine which pre-allocation quota has the least impact on the performance of the distributed file system. The pre-allocation quota with the least performance impact is then selected as the optimal pre-allocation quota for the current application scenario.

[0052] In step S3, after determining the optimal pre-allocated quota that meets the current application scenario, it is set into the distributed file system for application.

[0053] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota includes:

[0054] Different pre-request quotas are evaluated based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota.

[0055] In some embodiments of the present invention, evaluating different pre-request quotas based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota includes:

[0056] Obtain the data transfer request volume and pre-allocation request volume processed by the distributed file system under different pre-allocation quota scenarios;

[0057] The ratio of the data transmission request volume to the pre-requested quota volume is used as the score for the pre-requested quota;

[0058] By comparing the scores of different pre-application quotas, the pre-application quota with the highest score is selected as the optimal pre-application quota.

[0059] In this embodiment, when obtaining each pre-allocation quota application, the data transmission request and pre-allocation request workload processed by the distributed file system are specifically defined. The data transmission request refers to the data transmission request sent by the client to the distributed file system after obtaining the storage space allocated by the pre-allocation quota. That is, assuming a client sends a pre-allocation request to the metadata server of the distributed file system and obtains 100MB of storage space for writing data, the client needs to send a data transmission request to the distributed file system, and the data can only be transmitted after the distributed file system responds.

[0060] Therefore, in this embodiment, the impact of applying different pre-allocation quotas to the distributed file system is evaluated based on the workload of data transmission requests and pre-allocation requests. Specifically, the data transmission request volume and pre-allocation request volume corresponding to different pre-allocation quotas are obtained, and the data transmission request volume and pre-allocation request volume are divided to obtain the score of the corresponding pre-allocation quota. The pre-allocation quota with the highest score is selected as the optimal pre-allocation quota.

[0061] In some embodiments of the present invention, the number of data transmission requests processed by the distributed file system and the number of pre-request requests do not simply refer to the number of requests completed. In some embodiments, the number of data transmission requests and the number of pre-request requests refer to the time consumed by the distributed file system to complete the data transmission requests and the pre-request requests.

[0062] In some embodiments of the present invention, a threshold corresponding to the ratio of the data transfer request volume to the pre-request volume is compared. If the threshold is exceeded, the pre-request quota for the task is considered satisfactory. Ideally, the workload of the distributed file system in processing data transfer requests should be significantly higher than the workload in processing pre-requests. Frequent processing of pre-requests does not generate substantial storage services.

[0063] In some embodiments of the present invention, the method further includes:

[0064] The idle rate of the corresponding pre-requested quota is calculated based on the idle space of each client;

[0065] The final score of the pre-requested quota is obtained by subtracting the idle rate corresponding to the pre-requested quota from the score of the pre-requested quota.

[0066] By comparing the final scores of different pre-application quotas, the pre-application quota with the highest final score is selected as the optimal pre-application quota.

[0067] Furthermore, in this embodiment, to prevent the pre-allocated quota from increasing excessively—because in principle, the larger the pre-allocated quota, the less likely the client is to use it all, thus preventing subsequent pre-allocation requests—the most extreme case is that the directory file capacity is directly allocated to the corresponding client at once. This would result in multiple data transmission requests despite only generating a pre-allocation request, leading to an excessively high score and making it impossible to measure the quality of the pre-allocated quota.

[0068] Therefore, in this embodiment, the quality of pre-allocated quotas is further determined by the ratio of the free space after a client obtains the corresponding storage quota to the pre-allocated quota space. A suitable pre-allocated quota should balance whether the storage space obtained by the client is idle, while also preventing the client from frequently sending pre-allocated requests (if the pre-allocated quota is too small). Therefore, assuming that multiple clients obtain the corresponding pre-allocated quota, the ratio between the total free space of the multiple clients and the total pre-allocated quota is calculated, and this ratio is subtracted from the pre-allocated quota score.

[0069] It is important to note that since the idle rate ratio is a decimal between 0 and 1, while the score of the pre-request quota may be greater than 1 (which is generally the case), the idle rate and score need to be normalized to the same data domain. That is, the highest score of multiple pre-request quotas is equivalent to 1. Then, the score of each pre-request quota is compared with the highest score to obtain a ratio as the new score for each pre-request quota. Then, the idle rate of the pre-request quota corresponding to the new score is subtracted, and the resulting score is the final score.

[0070] In some embodiments of the present invention, generating different pre-allocated quotas in a distributed file system according to a predetermined method includes:

[0071] Obtain the total capacity of the distributed file system, and use a predetermined proportion of the total capacity as the maximum value of the pre-requested quota;

[0072] Determine the minimum value of the pre-request quota, and generate multiple pre-request quotas between the maximum and minimum values ​​of the pre-request quota with a predetermined step size.

[0073] In this embodiment, when generating pre-request quotas, to prevent the pre-request quotas from becoming infinitely large, the maximum value of the pre-request quota can be set to one-tenth or one-hundredth of the total capacity of the file directory. The maximum value of the pre-request quota is limited according to a certain proportion of the total capacity of the directory files. At the same time, a minimum value of the pre-request quota is set. Between the minimum and maximum values, a corresponding step size is set according to the required number of pre-request quotas to generate multiple pre-request quotas.

[0074] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota further includes:

[0075] Based on the file specifications of the data transmission request, the task volume of the corresponding pre-request quota is classified according to the data transmission request and the pre-request request. Under different categories, different pre-request quotas are evaluated based on the task volume of the data transmission request and the pre-request request to determine the optimal pre-request quota.

[0076] In this embodiment, to prevent abnormal data transfer request volumes due to varying data sizes (file sizes), which could lead to excessively large or small data transfer requests and consequently, inaccurate pre-allocated quota scores, data transfer requests are categorized based on file size. For file transfer requests of the same size, scores for different pre-allocated quotas are compared. Idle time can also be considered for further evaluation.

[0077] In some embodiments of the present invention, the method further includes:

[0078] The AI ​​model is trained based on the business operation data of the distributed file system under different pre-request quotas, and the corresponding pre-request quota is output based on the data transmission request based on the AI ​​model.

[0079] In this embodiment, data under different pre-application quotas are used to train an artificial intelligence model, which then provides the corresponding pre-application quota based on the specific application scenario.

[0080] Example:

[0081] Based on the Darknet53 deep residual network model, the current directory capacity, directory offset capacity, and directory quota capacity are used as inputs, and the pre-allocated quota value is used as the model output. The model is trained on a distributed file system cluster to obtain a calculation model that can provide different pre-allocated quota values ​​according to different capacity sizes in different directory scenarios.

[0082] Using the Darknet53 deep residual network model as an adaptive method for calculating pre-allocated quota values, and with the acquisition of a large amount of actual training data, it can accurately provide the error value of quota limit and actual final write capacity based on different directory capacity and actual read / write conditions, thereby reducing the number of IO operations and improving the reliability of the distributed file system.

[0083] Specific Implementation Methods of the Invention

[0084] This invention is based on the design and implementation of algorithms in a C++, Linux environment, and distributed system.

[0085] An adaptive directory quota pre-allocation method for distributed file systems. The specific implementation process is as follows:

[0086] (1) On the Linux operating system, with the write process of the distributed file system, under the premise of enabling directory quota, any write process or any operation that causes the capacity of the directory to increase is triggered in the target directory and its subdirectories. If the capacity offset is greater than the remaining value of the last pre-requested quota, the action of pre-requesting quota value is triggered.

[0087] (2) The steps for applying for the pre-application quota value are as follows:

[0088] The client sends a request to the MDS server, recording the current directory capacity, directory offset capacity, and total directory capacity quota. Using a DarKnet53 deep residual network pre-trained model, it outputs a generated single pre-request quota value. The original pre-request quota is added to the newly acquired pre-request quota. The directory's capacity quota information is updated and saved in the extended attributes of the directory file. If the new remaining quota is still less than the directory offset capacity, the pre-request quota process continues. Once the remaining quota is greater than the directory offset capacity, the data is updated, the remaining quota is subtracted from the directory offset capacity to obtain the new remaining directory capacity quota, and a pre-request success is returned.

[0089] (3) The action steps of the network pre-trained model are as follows:

[0090] Training process: Based on any deep learning training environment, including but not limited to deep learning frameworks such as PyTorch and TensorFlow, in a distributed storage file system, quota settings are manually executed and write operations are performed. The fixed value of the pre-allocated quota is adjusted based on experience and results. The current capacity of the directory, the directory offset capacity, the directory quota capacity, the error value between the quota limit and the actual final write capacity, the number of interactions with the MDS server, and multiple tests are conducted. An appropriate pre-allocated quota value is selected as the current capacity of the directory, the directory offset capacity, and the directory quota capacity as inputs to predict the true value of the output.

[0091] Collect data and create a dataset for training; the network output is the specific value of the quota pre-application.

[0092] (4) The steps to be taken after the pre-application quota value is completed are as follows:

[0093] 1. After successful pre-application, write operation is performed, data is written to the target directory, quota information is updated and recorded, metadata information such as directory capacity and modification time is updated, and quota usage of the target directory is statistically analyzed.

[0094] 2. If the write operation fails after pre-application, such as exceeding the current total quota limit of the directory, the quota application will fail and the quota information of the directory capacity will be rolled back to the directory capacity quota status before pre-application.

[0095] The present invention proposes a distributed file system directory quota optimization method, which analyzes the impact of different pre-allocation quotas on the distributed file system based on the operating data of the distributed file system under different pre-allocation quotas, finds the optimal pre-allocation quota, and applies the optimal pre-allocation quota to the distributed file system to improve the performance of the distributed file system.

[0096] like Figure 2 As shown, another aspect of the present invention proposes a distributed file system directory quota optimization system, comprising:

[0097] The pre-request quota generation module 1 is configured to generate different pre-request quotas in a distributed file system according to a predetermined method, and respond to the client's pre-request request with different pre-request quotas;

[0098] The pre-application quota evaluation module 2 is configured to evaluate the business operation data of the distributed file system under different pre-application quotas according to a predetermined strategy in order to obtain the optimal pre-application quota.

[0099] The pre-allocation quota implementation module 3 is configured to apply the optimal pre-allocation quota to the distributed file system in response to determining the optimal pre-allocation quota.

[0100] like Figure 3 As shown, another aspect of the present invention also provides a computer device comprising:

[0101] At least one processor 21; and

[0102] Memory 22 stores computer instructions 23 that can be executed on processor 21. When executed by processor 21, the instructions 23 implement a distributed file system directory quota optimization method, including:

[0103] Different pre-allocation quotas are generated in the distributed file system according to a predetermined method, and the client's pre-allocation request is responded to with different pre-allocation quotas;

[0104] The distributed file system's operational data under different pre-allocation quotas is evaluated according to a predetermined strategy to obtain the optimal pre-allocation quota.

[0105] In response to determining the optimal pre-allocated quota, the optimal pre-allocated quota is applied to the distributed file system.

[0106] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota includes:

[0107] Different pre-request quotas are evaluated based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota.

[0108] In some embodiments of the present invention, evaluating different pre-request quotas based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota includes:

[0109] Obtain the data transfer request volume and pre-allocation request volume processed by the distributed file system under different pre-allocation quota scenarios;

[0110] The ratio of the data transmission request volume to the pre-requested quota volume is used as the score for the pre-requested quota;

[0111] By comparing the scores of different pre-application quotas, the pre-application quota with the highest score is selected as the optimal pre-application quota.

[0112] In some embodiments of the present invention, the method further includes:

[0113] The idle rate of the corresponding pre-requested quota is calculated based on the idle space of each client;

[0114] The final score of the pre-requested quota is obtained by subtracting the idle rate corresponding to the pre-requested quota from the score of the pre-requested quota.

[0115] By comparing the final scores of different pre-application quotas, the pre-application quota with the highest final score is selected as the optimal pre-application quota.

[0116] In some embodiments of the present invention, generating different pre-allocated quotas in a distributed file system according to a predetermined method includes:

[0117] Obtain the total capacity of the distributed file system, and use a predetermined proportion of the total capacity as the maximum value of the pre-requested quota;

[0118] Determine the minimum value of the pre-request quota, and generate multiple pre-request quotas between the maximum and minimum values ​​of the pre-request quota with a predetermined step size.

[0119] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota further includes:

[0120] Based on the file specifications of the data transmission request, the task volume of the corresponding pre-request quota is classified according to the data transmission request and the pre-request request. Under different categories, different pre-request quotas are evaluated based on the task volume of the data transmission request and the pre-request request to determine the optimal pre-request quota.

[0121] In some embodiments of the present invention, the method further includes:

[0122] The AI ​​model is trained based on the business operation data of the distributed file system under different pre-request quotas, and the corresponding pre-request quota is output based on the data transmission request based on the AI ​​model.

[0123] like Figure 4 As shown, another aspect of the present invention provides a computer-readable storage medium 401, which stores a computer program 402. When executed by a processor, the computer program 402 implements a distributed file system directory quota optimization method, including:

[0124] Different pre-allocation quotas are generated in the distributed file system according to a predetermined method, and the client's pre-allocation request is responded to with different pre-allocation quotas;

[0125] The distributed file system's operational data under different pre-allocation quotas is evaluated according to a predetermined strategy to obtain the optimal pre-allocation quota.

[0126] In response to determining the optimal pre-allocated quota, the optimal pre-allocated quota is applied to the distributed file system.

[0127] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota includes:

[0128] Different pre-request quotas are evaluated based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota.

[0129] In some embodiments of the present invention, evaluating different pre-request quotas based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota includes:

[0130] Obtain the data transfer request volume and pre-allocation request volume processed by the distributed file system under different pre-allocation quota scenarios;

[0131] The ratio of the data transmission request volume to the pre-requested quota volume is used as the score for the pre-requested quota;

[0132] By comparing the scores of different pre-application quotas, the pre-application quota with the highest score is selected as the optimal pre-application quota.

[0133] In some embodiments of the present invention, the method further includes:

[0134] The idle rate of the corresponding pre-requested quota is calculated based on the idle space of each client;

[0135] The final score of the pre-requested quota is obtained by subtracting the idle rate corresponding to the pre-requested quota from the score of the pre-requested quota.

[0136] By comparing the final scores of different pre-application quotas, the pre-application quota with the highest final score is selected as the optimal pre-application quota.

[0137] In some embodiments of the present invention, generating different pre-allocated quotas in a distributed file system according to a predetermined method includes:

[0138] Obtain the total capacity of the distributed file system, and use a predetermined proportion of the total capacity as the maximum value of the pre-requested quota;

[0139] Determine the minimum value of the pre-request quota, and generate multiple pre-request quotas between the maximum and minimum values ​​of the pre-request quota with a predetermined step size.

[0140] In some embodiments of the present invention, evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota further includes:

[0141] Based on the file specifications of the data transmission request, the task volume of the corresponding pre-request quota is classified according to the data transmission request and the pre-request request. Under different categories, different pre-request quotas are evaluated based on the task volume of the data transmission request and the pre-request request to determine the optimal pre-request quota.

[0142] In some embodiments of the present invention, the method further includes:

[0143] The AI ​​model is trained based on the business operation data of the distributed file system under different pre-request quotas, and the corresponding pre-request quota is output based on the data transmission request based on the AI ​​model.

[0144] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0145] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0146] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0147] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0148] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of the different aspects of the invention as described above exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for optimizing directory quotas in a distributed file system, characterized in that, include: Different pre-allocation quotas are generated in the distributed file system according to a predetermined method, and the client's pre-allocation request is responded to with different pre-allocation quotas; The distributed file system's operational data under different pre-allocation quotas is evaluated according to a predetermined strategy to obtain the optimal pre-allocation quota. In response to determining the optimal pre-allocated quota, the optimal pre-allocated quota is applied to the distributed file system; The step of evaluating the operational data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota includes: The pre-request quotas are evaluated based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota. The process of evaluating different pre-request quotas based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota includes: Obtain the data transfer request volume and pre-allocation request volume processed by the distributed file system under different pre-allocation quota scenarios; The ratio of the data transmission request volume to the pre-requested quota volume is used as the score for the pre-requested quota; By comparing the scores of different pre-application quotas, the pre-application quota with the highest score is selected as the optimal pre-application quota.

2. The method according to claim 1, characterized in that, Also includes: The idle rate of the corresponding pre-requested quota is calculated based on the idle space of each client; The final score of the pre-requested quota is obtained by subtracting the idle rate corresponding to the pre-requested quota from the score of the pre-requested quota. By comparing the final scores of different pre-application quotas, the pre-application quota with the highest final score is selected as the optimal pre-application quota.

3. The method according to claim 1, characterized in that, The generation of different pre-allocated quotas in the distributed file system according to a predetermined method includes: Obtain the total capacity of the distributed file system, and use a predetermined proportion of the total capacity as the maximum value of the pre-requested quota; Determine the minimum value of the pre-request quota, and generate multiple pre-request quotas between the maximum and minimum values ​​of the pre-request quota with a predetermined step size.

4. The method according to claim 1, characterized in that, The step of evaluating the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota also includes: Based on the file specifications of the data transmission request, the task volume of the corresponding pre-request quota is classified according to the data transmission request and the pre-request request. Under different categories, different pre-request quotas are evaluated based on the task volume of the data transmission request and the pre-request request to determine the optimal pre-request quota.

5. The method according to claim 1, characterized in that, Also includes: The AI ​​model is trained based on the business operation data of the distributed file system under different pre-request quotas, and the corresponding pre-request quota is output based on the data transmission request based on the AI ​​model.

6. A distributed file system directory quota optimization system, characterized in that, include: A pre-request quota generation module is configured to generate different pre-request quotas in a distributed file system according to a predetermined method, and respond to the client's pre-request request with different pre-request quotas; A pre-allocation quota evaluation module is configured to evaluate the business operation data of the distributed file system under different pre-allocation quotas according to a predetermined strategy in order to obtain the optimal pre-allocation quota. A pre-allocation quota implementation module is configured to apply the optimal pre-allocation quota to the distributed file system in response to determining the optimal pre-allocation quota. The step of evaluating the operational data of the distributed file system under different pre-allocation quotas according to a predetermined strategy to obtain the optimal pre-allocation quota includes: The pre-request quotas are evaluated based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota. The process of evaluating different pre-request quotas based on the workload of data transmission requests and pre-request requests to determine the optimal pre-request quota includes: Obtain the data transmission request volume and pre-request volume processed by the distributed file system under different pre-request quota scenarios; use the ratio of the data transmission request volume to the pre-request volume as the score of the pre-request quota; compare the scores of different pre-request quotas, and take the pre-request quota with the highest score as the optimal pre-request quota.

7. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-5.

Images