Quota management method, computer device, storage medium and program product

Abstract

The present application relates to a quota management method, computer equipment, storage medium and program product, and belongs to the field of distributed storage technology. The method includes: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; obtaining quota identification authentication information from a metadata server using a client based on the quota prediction value within the target time window; generating a write request based on the quota identification authentication information, and sending the write request to a target storage node using a client; the target storage node allocates storage resources according to the quota identification authentication information corresponding to the write request, and returns it to the client. The present application can adapt to different business loads, and by predicting quotas, it can reduce the performance overhead of the distributed storage system caused by updating quota messages, thereby ensuring the global consistency, accuracy and performance of quota management.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a quota management method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of cloud computing, big data and the popularization of smart devices, the amount of data and the type of data are growing rapidly, and traditional storage systems can no longer meet the large-scale and diversified data storage needs. The rise of big data and cloud computing technologies has provided more application scenarios for distributed storage systems and promoted the development of distributed storage system technology. A distributed storage system refers to a system that stores data on multiple independent nodes and manages it in a distributed manner. It aims to provide a high-performance, high-reliability and scalable storage solution. The design concept of a distributed storage system is based on scalability and fault tolerance. It adopts a distributed architecture, slices data and distributes it in the form of objects on multiple nodes, and ensures that the capacity and load of each node are balanced. It uses a technology that accesses and manages data through the network and provides a unified storage interface. Among them, high reliability: data redundancy and automatic fault recovery mechanisms are used to ensure data reliability and persistence; high scalability: using a decentralized design architecture, the distributed storage system cluster can be dynamically expanded horizontally to support hundreds or even thousands of nodes. The failure of a single storage server node will not affect the unavailability of the entire distributed storage cluster. It supports dynamic expansion to add or delete storage server nodes, and can automatically relocate data objects in the cluster to balance the storage space utilization between storage servers; unified interface: provides a unified file system, object storage and block storage interface to facilitate users to access and manage data.

[0003] In distributed storage systems, quota management is an important technology used to manage and limit the resources of distributed storage systems, such as disk space and number of files. Through file system quota management, system administrators can limit the use of storage system resources by users or applications to ensure fair allocation and effective use of system resources. Through quota restrictions and management, excessive use of system resources can be avoided and system performance and stability can be improved. Effective management of storage resources can help the system provide better services under limited resources. Some industry standards and regulations may require monitoring and limiting the use of storage system resources to ensure data security and compliance.

[0004] However, as the demand for quota management in the industry increases, quota management will introduce some additional performance overhead, including regular checks on the storage usage of users or directories. These additional checks will increase the load and overhead of the distributed storage system. Improper setting of the check period will lead to inaccurate quota statistics. The distributed storage system needs to send notifications when quota events occur (such as quota usage approaching the upper limit, over-usage, etc.). These notification mechanisms will introduce additional communication overhead and processing overhead.

[0005] In the related prior art, the distributed storage file system implements quota management by setting extended attributes (ExtendedAttributes, xattrs). This extended attribute is a key / value storage format. When each client performs a write operation, the xattr attribute of the directory will be updated; since the distributed storage system is multi-node and scalable, there are scenarios where multiple nodes concurrently operate the same file system quota, and metadata services are required to make unified decisions. The metadata service aggregates the xattr extended attribute information from multiple nodes, and distributes it to each node after unified processing; in a single-node environment, the system can only accurately track the resource consumption of the local machine, and in the face of cross-node directory operations, if there is no unified management mechanism, it is easy to cause inconsistencies and errors in capacity statistics.

[0006] In order to meet the global consistency and accuracy of the quota management of the entire distributed storage system, the metadata service needs to continuously synchronize and regularly detect xattr attribute messages. Because of this, quota management increases the performance overhead of the storage system under high load. When the file quota usage is large and the business load is high, quota management will cause the metadata service to be overloaded, and then unable to handle normal storage IO (Input / Output), causing application blocking. In addition, using the metadata service as the center to summarize and broadcast quota information management, while ensuring global consistency, it also introduces potential delay problems. For example, when multiple concurrent clients operate on files and directories, the metadata service will not summarize and broadcast quota information in a timely manner, which will lead to inaccurate quota statistics or even over-limit problems. Therefore, how to ensure that quota management does not introduce too much performance overhead when implementing quota management on distributed storage systems and ensure the effective use of system resources is an urgent problem to be solved. Summary of the invention

[0007] Based on this, it is necessary to provide a quota management method, apparatus, computer device and storage medium that can ensure that quota management does not introduce excessive performance overhead while ensuring that system resources can be used effectively in response to the above technical problems.

[0008] In a first aspect, a quota management method is provided, which is applied to a distributed storage system, wherein the distributed storage system includes at least a client, a metadata server, and multiple storage nodes, and the quota management method includes: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; obtaining quota identifier authentication information from the metadata server using the client based on the quota prediction value within the target time window; generating a write request based on the quota identifier authentication information, and sending the write request to a target storage node using the client; the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns the storage resources to the client.

[0009] Optionally, the method for generating the time series identifier includes: in response to detecting that a client has issued a write request, identifying the write request based on the issuance time of the write request; and generating a time series identifier based on the time identifiers of multiple write requests within a target time period.

[0010] Optionally, based on the time series identifier and the quota prediction mechanism, determining the quota prediction value within the target time window includes: determining the target time window corresponding to the current time based on the time series identifier; constructing multiple prediction units based on the number of clients, and initializing the weight of each prediction unit; obtaining the quota prediction value of each prediction unit in the target time window, and determining the quota prediction value within the target time window based on the weight of each prediction unit.

[0011] Optionally, before using the client to obtain quota identification authentication information from the metadata server, the method further includes: constructing a quota identification authentication request based on the client, wherein the quota identification authentication request at least includes a quota prediction value within a target time window.

[0012] Optionally, based on the quota prediction value within the target time window, using the client to obtain quota identification authentication information from the metadata server includes: based on a non-real-time asynchronous communication mechanism, using the client to send a constructed quota identification authentication request to the metadata server; when the metadata server receives the quota identification authentication request, it constructs quota identification authentication information and returns the quota identification authentication information to the client.

[0013] Optionally, the method for constructing the quota identification authentication information includes: determining the expiration time of the quota identification authentication information based on the target time window; determining the incremental sequence number based on the number of quota identification authentication requests received by the metadata server; determining the quota capacity based on the quota prediction value; generating a valid credential consisting of an encrypted byte sequence according to the application time and expiration time of the quota identification authentication information, the quota capacity and the incremental sequence number, and defining the valid credential as the quota identification authentication information.

[0014] Optionally, the method for generating the write request includes: acquiring quota identification authentication information; and generating the write request based on the quota identification authentication information.

[0015] Optionally, after the target storage node receives the write request, the method further includes: detecting the quota identification authentication information of the write request to generate a detection result; in response to detecting that the detection result meets the preset standard, allocating storage resources according to the quota capacity determined based on the quota prediction value, and returning the storage resources to the client; in response to detecting that the detection result does not meet the preset standard, returning a prompt message to the user end.

[0016] Optionally, the distributed storage system also includes multiple interfaces, and the method also includes: sending a quota identification authentication request based on the first interface, and responding to the quota identification authentication request based on the second interface to generate quota identification authentication information; detecting the quota identification authentication information based on the third interface to generate a detection result; in response to detecting that the detection result meets a preset standard, providing the allocated storage resources to the client based on the fourth interface, and maintaining the allocated storage resources based on the fifth interface.

[0017] Optionally, after returning the allocated storage resources to the client, the method further includes: obtaining real-time storage resource usage within a target time window; comparing the real-time storage resource usage within the target time window with the amount of allocated storage resources to obtain a comparison result; and adjusting the weight of each prediction unit based on the comparison result.

[0018] Optionally, the real-time storage resource usage within the target time window is compared with the resource amount of the allocated storage resources, and the comparison result includes: obtaining the difference between the real-time storage resource usage and the resource amount of the allocated storage resources; comparing the difference with a preset threshold; in response to detecting that the difference is greater than the preset threshold, obtaining a first comparison result; in response to detecting that the difference is less than or equal to the preset threshold, obtaining a second comparison result.

[0019] Optionally, adjusting the weight of each prediction unit based on the comparison result includes: in response to detecting that the comparison result is a first comparison result, calculating and determining the weight of each prediction unit based on a first weight adjustment algorithm; and adjusting the weight of each prediction unit according to the calculated weight of each prediction unit.

[0020] Optionally, adjusting the weight of each prediction unit based on the comparison result includes: in response to detecting that the comparison result is a second comparison result, calculating and determining the weight of each prediction unit based on a second weight adjustment algorithm; and adjusting the weight of each prediction unit according to the calculated weight of each prediction unit.

[0021] Optionally, before calculating and determining the weight of each prediction unit based on the first weight adjustment algorithm, the method further includes: obtaining a ratio of a prediction value of the i-th prediction unit in the t+1 prediction test to the sum of the prediction values ​​of all prediction units in the t+1 prediction test, and the calculation formula is:

[0022] ;in, Represents the ratio, Indicates The predicted value of the prediction test, Indicates the number of prediction units.

[0023] Optionally, calculating and determining the weight of each prediction unit based on a first weight adjustment algorithm includes: calculating and determining the weight of each prediction unit based on the ratio and the first weight adjustment algorithm, wherein the first weight adjustment algorithm includes:

[0024] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0025] Optionally, before calculating and determining the weight of each prediction unit based on the second weight adjustment algorithm, the method further includes: calculating the absolute value of the difference between the actual value and the predicted value to obtain a deviation value Based on the deviation value , calculate and determine the ratio corresponding to the deviation value, and the calculation formula is:

[0026] ;in, Indicates the ratio corresponding to the deviation value, Indicates the deviation value, Indicates the number of prediction units.

[0027] Optionally, calculating and determining the weight of each prediction unit based on the second weight adjustment algorithm includes: calculating and determining the weight of each prediction unit based on the ratio corresponding to the deviation value and the second weight adjustment algorithm, wherein the second weight adjustment algorithm includes:

[0028] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0029] In a second aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; obtaining quota identifier authentication information from the metadata server using the client based on the quota prediction value within the target time window; generating a write request based on the quota identifier authentication information, and sending the write request to a target storage node using the client; the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns the resources to the client.

[0030] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; obtaining quota identifier authentication information from the metadata server using the client based on the quota prediction value within the target time window; generating a write request based on the quota identifier authentication information, and sending the write request to a target storage node using the client; the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns the resources to the client.

[0031] In a fourth aspect, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the following steps are implemented: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; obtaining quota identification authentication information from the metadata server using the client based on the quota prediction value within the target time window; generating a write request based on the quota identification authentication information, and sending the write request to a target storage node using the client; the target storage node allocates storage resources according to the quota identification authentication information corresponding to the write request, and returns the resources to the client.

[0032] The above-mentioned quota management method, computer device, storage medium and program product, the method includes: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; based on the quota prediction value within the target time window, using the client to obtain quota identifier authentication information from the metadata server; based on the quota identifier authentication information, generating a write request, and using the client to send the write request to a target storage node; the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns it to the client. The present application can adapt to different business loads, and by predicting quotas, it reduces the performance overhead problem of the distributed storage system caused by updating quota messages, thereby ensuring the global consistency, accuracy and performance of quota management. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The figure is a flowchart of a quota management method in an embodiment.

[0034] Figure 2 FIG. 4 is another flow chart of a quota management method in an embodiment.

[0035] Figure 3 The figure is a schematic diagram of the overall architecture of a distributed storage system of a quota management method in an embodiment.

[0036] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0038] It should be understood that in the description of the present application, unless the context clearly requires otherwise, words such as "include", "comprises", and the like throughout the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as including but not limited to.

[0039] It should also be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, "plurality" means two or more.

[0040] It should be noted that the terms "S1", "S2", etc. are only used for the purpose of describing the steps, and do not specifically refer to the order or sequence, nor are they used to limit the present application. They are only for the convenience of describing the method of the present application, and cannot be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present application.

[0041] To solve the problems of the prior art, the present application provides a quota management method, apparatus, computer equipment and storage medium. The quota authentication identification information and quota prediction mechanism can effectively prevent the impact on the performance of the distributed storage system during the quota management process, control storage resource consumption and prevent individual users or applications from excessively occupying shared storage resources, and solve the problem of storage IO performance degradation caused by quota management. In dynamic workloads, the quota weights of multiple prediction units are adjusted to improve the accuracy of the quota, while reducing the pressure on synchronizing metadata services and updating quota information.

[0042] In one embodiment, Figure 1 and Figure 2 As shown, a quota management method is provided, which is applied to a distributed storage system. The quota management method includes steps S1-S4.

[0043] S1: Obtain a time series identifier corresponding to a client write request, and determine a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism.

[0044] It should be noted that if Figure 3 As shown in FIG. 1 , it is an overall architecture diagram of a distributed storage system. The distributed storage system includes at least a client (i.e., a client application), a metadata server (i.e., a metadata and monitoring service cluster), and multiple storage nodes, which specifically include the following key components: (1) A unified, self-controlled, and scalable distributed storage consistency management system that performs data transmission through a file system interface: This component is the core component of a distributed storage cluster. It provides distributed storage services in the form of objects. In this component, data is stored as objects. Each object has a unique identifier and related data. It is responsible for storing objects in a distributed manner on each node of the storage cluster and provides functions such as data replication, recovery, and load balancing to ensure data reliability and high-performance access; (2) Storage pool: A distributed storage cluster consists of multiple storage nodes, each of which can contain multiple hard disks. Or storage device, the storage pool divides the storage resources to form different pools to meet different storage needs. In the storage pool, the client data is stored by slicing, which is fixed to 4MB objects by default. The objects are grouped, and the group id is the value modulo Hash (object x) and the number of groups, and finally stored in the underlying device; (3) Controllable, scalable, distributed data balanced placement algorithm: This component evenly distributes data objects on each node of the storage cluster to avoid data hotspots and improve system performance; (4) Metadata and monitoring service cluster: The metadata cluster is responsible for managing the metadata information of the file system. In this application, it serves as the main service for quota management, including the attributes and location of files and directories. The monitoring service cluster is responsible for monitoring the status and configuration information of the cluster, which includes monitoring and maintaining the status information, configuration information, health status, etc. of the cluster to ensure the consistency and availability of the cluster.

[0045] In some specific embodiments, the method for generating the time series identifier includes: in response to detecting that a write request is issued by a client, identifying the write request based on the issuance time of the write request; generating a time series identifier based on the time identifiers of multiple write requests within a target time period, that is, identifying the write requests issued by the client application through a changing time series for predicting the quota of the target time window, wherein the target time period can be set according to actual needs, such as 1 day, 2 days, etc.

[0046] In some specific implementations, based on the time series identifier and the quota prediction mechanism, determining the quota prediction value within the target time window includes: based on the time series identifier, determining the target time window corresponding to the current time, that is, the time period for which prediction is required, such as if the current time is 12 o'clock, if the time series identifier is segmented into 1 hour, then the target time window is from 12 o'clock to 13 o'clock; based on the number of clients, constructing multiple prediction units, and initializing the weight of each prediction unit, that is, each client is correspondingly provided with a prediction unit, and the weight of each client prediction unit is initialized to ,in Represents the number of prediction units; obtains the quota prediction value of each prediction unit in the target time window, and determines the quota prediction value in the target time window based on the weight of each prediction unit, wherein the quota prediction value of each prediction unit in the target time window can be predicted by the predictor corresponding to each prediction unit, and each predictor stores the storage resource usage corresponding to multiple time periods. For example, if the target time window is the storage resource usage from 12:00 to 13:00 on Wednesday, then according to the time series, the quota prediction value from 12:00 to 13:00 every Wednesday can be known, and the initial weight of each prediction unit is , the initial weight and the quota prediction value of each prediction unit are multiplied and added to obtain the quota prediction value within the target time window.

[0047] In the above implementation, the write request of the client application is identified by a changing time series, and the quota is predicted based on the sequence, which can reduce the number of quota identification authentication messages and reduce the transmission overhead of the distributed storage system quota message between the client and the metadata server, and the metadata server and each node of the distributed storage.

[0048] S2: Based on the quota prediction value within the target time window, using the client to obtain quota identification authentication information from the metadata server.

[0049] It should be noted that non-real-time asynchronous communication is adopted between the client and the metadata server and the storage node, that is, the sender and the receiver have no synchronization timing constraints during the communication process. They can operate independently of each other and only need to communicate with the metadata server periodically. They are independent of storage IO and do not need to obtain quota authorization from the metadata server before writing every time a write request is generated. This method can reduce the load pressure of the metadata service, improve the performance of the client business, and ensure the global consistency of quota management.

[0050] In some specific implementations, based on the quota prediction value within the target time window, obtaining the quota identification authentication information from the metadata server using the client includes: based on a non-real-time asynchronous communication mechanism, using the client to send a constructed quota identification authentication request to the metadata server, wherein the identification authentication is a field designed in the quota management module structure, stored in the memory, and is different from the metadata extended attribute xattr, and is intended to prevent excessive quota information from affecting metadata queries; when the metadata server receives the quota identification authentication request, it constructs the quota identification authentication information and returns the quota identification authentication information to the client, that is, it allocates quota identification authentication information according to the information in the quota identification authentication field. The storage resource is stored and returned to the client user, wherein the quota identification authentication information is the storage resource requested by the client, that is, the valid certificate for the assigned quota, and is composed of an encrypted byte sequence, including the quota application time, expiration time, quota capacity and an incremental serial number, wherein the application time is the quota setting time, the expiration time is the identification authentication validity period, the quota capacity is the storage space that needs to be applied for, and the incremental serial number sets a counter according to the quota setting request received by the storage system, and cumulatively increases the integer 1, thereby ensuring the uniqueness of each quota identification authentication information and preventing conflicts with other client quota authentication identification information; at the same time, the quota management module and the storage node will regularly detect the quota identification authentication information and update the quota information in a timely manner.

[0051] In some specific implementations, the method for constructing the quota identification authentication information includes: determining the expiration time of the quota identification authentication information based on the target time window, that is, defining the last time point in the target time window as the expiration time; determining the incremental sequence number based on the number of quota identification authentication requests received by the metadata server; determining the quota capacity based on the quota prediction value; generating a valid credential composed of an encrypted byte sequence according to the application time and expiration time of the quota identification authentication information, the quota capacity and the incremental sequence number, and defining the valid credential as the quota identification authentication information.

[0052] In some specific implementations, before using the client to obtain quota identification authentication information from the metadata server, the method also includes: based on the client, constructing a quota identification authentication request, the quota identification authentication request at least includes a quota prediction value within a target time window to determine the quota capacity.

[0053] In the above implementation, the storage space required between the current request and the next request is predicted through the recent storage resource consumption history of the distributed client, and the quota identification authentication message is updated to the metadata server. Through the obtained quota identification authentication information, the client can use any available distributed storage system resources without having to obtain quota authorization from the metadata server multiple times. This can reduce the load pressure of the metadata server, improve the performance of the client business, and at the same time ensure the global consistency of quota management.

[0054] S3: Generate a write request based on the quota identification authentication information, and use the client to send the write request to the target storage node.

[0055] It should be noted that the write request is sent by a client user to apply for storage resources from a target storage node.

[0056] In some specific implementations, the method for generating the write request includes: obtaining quota identification authentication information; and generating the write request based on the quota identification authentication information.

[0057] After generating a write request, the write request is sent to a target storage node based on a non-real-time asynchronous communication mechanism. In some specific embodiments, after the target storage node receives the write request, the method further includes: detecting the quota identification authentication information of the write request and generating a detection result, that is, the storage node verifies the quota identification authentication information; in response to detecting that the detection result meets a preset standard, allocating storage resources according to the quota capacity determined based on the quota prediction value, and returning the resources to the client, wherein the preset standard is successful verification; in response to detecting that the detection result does not meet the preset standard, returning a prompt message to the user end, wherein the prompt message may be information such as unsuccessful verification and inability to allocate resources.

[0058] In the above implementation, based on the non-real-time asynchronous communication mechanism, the sender and the receiver can operate independently of each other during the communication process, thereby improving the operating performance of the distributed storage system.

[0059] S4: The target storage node allocates storage resources according to the quota identification authentication information corresponding to the write request, and returns the storage resources to the client.

[0060] It should be noted that the amount of allocated storage resources is determined by the quota capacity in the quota identification authentication information.

[0061] In some specific embodiments, the distributed storage system also includes multiple interfaces, among which the number of message interfaces is 5, and the method also includes: sending a quota identification authentication request based on the first interface, and responding to the quota identification authentication request based on the second interface to generate quota identification authentication information; detecting the quota identification authentication information based on the third interface to generate a detection result; in response to detecting that the detection result meets the preset standard, providing the allocated storage resources to the client based on the fourth interface, and maintaining the allocated storage resources based on the fifth interface.

[0062] Specifically, in a distributed storage system, the first interface is QuotaAuthRequest, which refers to a quota identification authentication request, the second interface is QuotaAuthReply, which refers to a quota identification authentication response, the third interface is OSDcheck, which refers to a storage node device check, the fifth interface is EpochSync, which refers to timing synchronization, and the fourth interface is OSDQuotaAuthReply, which refers to a storage node device quota authentication response. The client first uses the QuotaAuthRequest interface to send a quota identification authentication request message, and the metadata server uses the QuotaAuthRequest message to provide authentication to the requesting client, wherein a quota prediction mechanism is set in the metadata server. This application tracks and records the total authentication information issued to the client during a specified period, and the client carries the authentication information to the OSD device (storage node, such as the primary osd node 1 and the secondary osd node 2) and sends an OSDcheck message for authentication verification. If the authentication is valid, it responds to the client through OSDQuotaAuthReply and provides storage resources. The OSD maintains the total quota allocated to each client through the EpochSync message in each Epoch cycle.

[0063] In some specific embodiments, after returning the allocated storage resources to the client, the method further includes: obtaining real-time storage resource usage within a target time window; comparing the real-time storage resource usage within the target time window with the amount of allocated storage resources to obtain a comparison result; and adjusting the weight of each prediction unit based on the comparison result.

[0064] In some specific embodiments, the real-time storage resource usage within the target time window is compared with the allocated storage resource amount, and the comparison result obtained includes: obtaining the difference between the real-time storage resource usage and the allocated storage resource amount, wherein the difference is calculated as a value obtained by subtracting the allocated storage resource amount from the real-time storage resource usage; comparing the difference with a preset threshold, wherein the preset threshold can be set according to actual needs, such as the preset threshold is 0; in response to detecting that the difference is greater than the preset threshold, obtaining a first comparison result, i.e., a prediction error; in response to detecting that the difference is less than or equal to the preset threshold, obtaining a second comparison result, i.e., a correct prediction.

[0065] In some specific embodiments, adjusting the weight of each prediction unit based on the comparison result includes: in response to detecting that the comparison result is a first comparison result, calculating and determining the weight of each prediction unit based on a first weight adjustment algorithm; adjusting the weight of each prediction unit according to the calculated weight of each prediction unit; in response to detecting that the comparison result is a second comparison result, calculating and determining the weight of each prediction unit based on a second weight adjustment algorithm; and adjusting the weight of each prediction unit according to the calculated weight of each prediction unit.

[0066] In some specific embodiments, before calculating and determining the weight of each prediction unit based on the first weight adjustment algorithm, the method further includes: obtaining a ratio of a prediction value of the i-th prediction unit in the t+1 prediction test to the sum of the prediction values ​​of all prediction units in the t+1 prediction test, and the calculation formula is:

[0067] ;in, Represents the ratio, Indicates The predicted value of the prediction test, Indicates the number of prediction units.

[0068] In some specific implementations, calculating and determining the weight of each prediction unit based on the first weight adjustment algorithm includes: calculating and determining the weight of each prediction unit based on the ratio and the first weight adjustment algorithm, the first weight adjustment algorithm including:

[0069] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0070] In some specific implementations, before calculating and determining the weight of each prediction unit based on the second weight adjustment algorithm, the method further includes: calculating the absolute value of the difference between the actual value and the predicted value to obtain a deviation value Based on the deviation value , calculate and determine the ratio corresponding to the deviation value, and the calculation formula is:

[0071] ;in, Indicates the ratio corresponding to the deviation value, Indicates the deviation value, Indicates the number of prediction units.

[0072] In some specific implementations, calculating and determining the weight of each prediction unit based on the second weight adjustment algorithm includes: calculating and determining the weight of each prediction unit based on the ratio corresponding to the deviation value and the second weight adjustment algorithm, the second weight adjustment algorithm including:

[0073] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0074] Specifically, in an embodiment of the present application, the sum of write operations performed by the client within a period of time is determined through a quota prediction mechanism. Considering that the quota prediction mechanism may fluctuate under different business load conditions or in different time periods, the time series data may not have linear stability, which may lead to inaccurate prediction results. Based on this, the present application proposes a dynamic and adaptive prediction method, which uses multiple prediction units to process dynamically changing workloads, that is, multiple prediction units use a weighted window method to perform quota prediction. This method dynamically captures the client write request time series, and then predicts the sum of the storage resource space allocation for write requests in a subsequent time window. This method can quickly identify changes in client write request data. When an error occurs in the prediction result or the prediction window time ends, this prediction test ends. The initial weight of a single prediction unit is expressed as , that is, in the predictors, and the test period is The prediction confidence level is the degree of certainty of the prediction algorithm model in its single prediction result, expressed as a probability value, ranging from 0% to 100%.

[0075] Furthermore, the present application predicts each prediction unit within a fixed time window, and then uses the standard weighting of the prediction unit for linear averaging. Finally, the product of each prediction unit weight and a single prediction value is added to calculate the actual prediction value. Whether a prediction error occurs is determined based on the above comparison result. When a prediction error occurs, the weight of each prediction unit is updated. Assuming that the quota identification authentication information expires before the target time window, the present application will readjust the weight of a single prediction unit and predict each prediction unit in the remaining window period. Specifically: When a prediction error occurs, based on the first weight adjustment algorithm, the weight of a single prediction unit is readjusted. First, the weight vector is defined. , when the prediction unit of each client has an incorrect prediction result, that is, when the business load bursts in the time window, the write request time series suddenly increases, causing the number of write requests passing to exceed the allowed quota value, after the end of each target time window, for each incorrect prediction, the weight of the prediction unit is updated according to the accuracy, and adjusted through the following algorithm:

[0076] ; .

[0077] In case of a prediction error, first calculate The value of is the ratio of the predicted value of the i-th prediction unit in the t+1 prediction test to the sum of the predicted values ​​of all prediction units in the t+1 prediction test. is the number of prediction units, that is, the number of all prediction units corresponding to the number of client requests in distributed storage, the normalization factor The function is to map the data to a specific interval or range, making the data comparable and interpretable. Responsible for predicting how fast the unit will be penalized for a given loss, When the value of is large, the weight of the prediction unit factor can be quickly reduced to zero, and the weight is rescaled. The distribution of the weight vector is linearly uniform, that is .

[0078] When the prediction is correct, the weights are updated at the end of the target time window based on the second weight adjustment algorithm. First, the weight vector is defined When the prediction result of each client's prediction unit is correct, that is, it does not exceed the allowed quota value, it is updated through the following algorithm: At the end of each prediction window, this application updates the prediction value by calculating the deviation As the difference between the actual and predicted values, It is the difference between the actual value and the predicted value (a positive number), where the actual value is the actual sum calculated at the last window time, the predicted value is the predicted value at the last window time, the ratio corresponding to the deviation value is determined by the deviation value, and the weight of the i-th prediction unit updated by the deviation is calculated.

[0079] In the above implementation, based on the quota weight adjustment algorithm of multiple prediction units, the weight of each prediction unit is adjusted according to the correctness of the prediction, thereby improving the accuracy of the quota in dynamic workloads, while reducing the pressure on the metadata server to synchronize and update quota information, effectively preventing the impact of the quota management process on the performance of the distributed storage system, controlling storage resource consumption and preventing individual users or applications from excessively occupying shared storage resources, and solving the problem of storage IO performance degradation caused by quota management.

[0080] In the above-mentioned quota management method, the method includes: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; based on the quota prediction value within the target time window, using the client to obtain quota identifier authentication information from the metadata server; based on the quota identifier authentication information, generating a write request, and using the client to send the write request to a target storage node; the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns it to the client. The quota identifier authentication method of the present application can effectively solve the problem of high metadata service load pressure caused by quota management, resulting in storage IO performance degradation and jitter, affecting the IO throughput of the application, and through quotas The prediction method adjusts and updates the weights, which can reduce the number of quota update messages sent by the quota management module to the metadata management server, so as to effectively solve the problem of periodically checking the storage usage of users or directories during quota usage. These additional checks will increase the load and overhead of the distributed storage system, and improper setting of the inspection cycle will lead to inaccurate quota statistics. The present application can adapt to different business loads and reduce the performance overhead of the distributed storage system caused by the quota message update through the quota prediction method, thereby ensuring the global consistency, accuracy and performance of quota management. The present application can effectively solve the quota usage in emergencies in dynamic workloads by predicting the total record of write requests within a period of time, avoiding the phenomenon of inaccurate quotas and overages, and reducing communication overhead and processing overhead.

[0081] Specifically, the present application has the following advantages: 1. Performance: Based on the same hardware resources, in dynamic workloads, unnecessary quota management performance overhead can be avoided, solving the load pressure on the metadata server; 2. Stability: The quota management method and prediction method of the present application are implemented in a structure of the metadata server, stored in the memory device, decoupled from the distributed storage system module, transparent to the business client, and stable; 3. Security: The components and method modules of the present application are encapsulated and processed, which is secure; 4. Low cost: The quota management and prediction algorithm and structured attribute information storage form proposed in the present application can improve the competitiveness and maintenance cost of distributed file system storage; 5. Compatibility: The quota management and prediction method of the embodiment of the present application is portable and universal on different hardware devices, especially compatible with scalable distributed clusters.

[0082] It should be understood that although Figure 1-Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0083] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a quota management method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0084] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0085] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: S1: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; S2: obtaining quota identifier authentication information from the metadata server using the client based on the quota prediction value within the target time window; S3: generating a write request based on the quota identifier authentication information, and sending the write request to a target storage node using the client; S4: the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns the storage resources to the client.

[0086] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in response to detecting that the client issues a write request, the write request is identified based on the issuance time of the write request; based on the time identification of multiple write requests within the target time period, a time series identification is generated.

[0087] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the time series identifier, determining the target time window corresponding to the current time; based on the number of clients, constructing multiple prediction units and initializing the weight of each prediction unit; obtaining the quota prediction value of each prediction unit in the target time window, and determining the quota prediction value within the target time window based on the weight of each prediction unit.

[0088] In one embodiment, when the processor executes the computer program, the following steps are further implemented: based on the client, constructing a quota identification authentication request, wherein the quota identification authentication request at least includes a quota prediction value within a target time window.

[0089] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on a non-real-time asynchronous communication mechanism, using the client to send a constructed quota identification authentication request to the metadata server; when the metadata server receives the quota identification authentication request, it constructs quota identification authentication information and returns the quota identification authentication information to the client.

[0090] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the target time window, determining the expiration time of the quota identification authentication information; based on the number of quota identification authentication requests received by the metadata server, determining the incremental sequence number; based on the quota prediction value, determining the quota capacity; based on the application time and expiration time of the quota identification authentication information, the quota capacity and the incremental sequence number, generating a valid credential composed of an encrypted byte sequence, and defining the valid credential as the quota identification authentication information.

[0091] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining quota identification authentication information; and generating the write request based on the quota identification authentication information.

[0092] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the quota identification authentication information of the write request is detected to generate a detection result; in response to detecting that the detection result meets the preset standard, storage resources are allocated according to the quota capacity determined based on the quota prediction value, and returned to the client; in response to detecting that the detection result does not meet the preset standard, prompt information is returned to the user end.

[0093] In one embodiment, when the processor executes the computer program, the following steps are also implemented: sending a quota identification authentication request based on the first interface, and responding to the quota identification authentication request based on the second interface to generate quota identification authentication information; detecting the quota identification authentication information based on the third interface to generate a detection result; in response to detecting that the detection result meets a preset standard, providing the allocated storage resources to the client based on the fourth interface, and maintaining the allocated storage resources based on the fifth interface.

[0094] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the real-time storage resource usage within the target time window; comparing the real-time storage resource usage within the target time window with the resource amount of the allocated storage resources to obtain a comparison result; and adjusting the weight of each prediction unit based on the comparison result.

[0095] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the difference between the real-time storage resource usage and the allocated storage resource usage; comparing the difference with a preset threshold; in response to detecting that the difference is greater than the preset threshold, obtaining a first comparison result; in response to detecting that the difference is less than or equal to the preset threshold, obtaining a second comparison result.

[0096] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in response to detecting that the comparison result is the first comparison result, based on the first weight adjustment algorithm, the weight of each prediction unit is calculated and determined; according to the weight of each prediction unit determined by the calculation, the weight of each prediction unit is adjusted.

[0097] In one embodiment, when the processor executes the computer program, the following steps are further implemented: in response to detecting that the comparison result is the second comparison result, based on the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined; according to the weight of each prediction unit determined by the calculation, the weight of each prediction unit is adjusted.

[0098] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining a ratio of a prediction value of the i-th prediction unit in the t+1 prediction test to the sum of the prediction values ​​of all prediction units in the t+1 prediction test, the calculation formula of which is:

[0099] ;in, Represents the ratio, Indicates The predicted value of the prediction test, Indicates the number of prediction units.

[0100] In one embodiment, when the processor executes the computer program, the following steps are further implemented: based on the ratio and the first weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the first weight adjustment algorithm includes:

[0101] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0102] In one embodiment, when the processor executes the computer program, the following steps are further implemented: calculating the absolute value of the difference between the actual value and the predicted value to obtain a deviation value Based on the deviation value , calculate and determine the ratio corresponding to the deviation value, and the calculation formula is:

[0103] ;in, Indicates the ratio corresponding to the deviation value, Indicates the deviation value, Indicates the number of prediction units.

[0104] In one embodiment, when the processor executes the computer program, the following steps are further implemented: based on the ratio corresponding to the deviation value and the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the second weight adjustment algorithm includes:

[0105] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0106] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: S1: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; S2: based on the quota prediction value within the target time window, using the client to obtain quota identifier authentication information from the metadata server; S3: based on the quota identifier authentication information, generating a write request, and using the client to send the write request to a target storage node; S4: the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns the storage resources to the client.

[0107] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to detecting that a client issues a write request, identifying the write request based on the issuance time of the write request; and generating a time series identifier based on the time identifiers of multiple write requests within a target time period.

[0108] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: based on the time series identifier, determining the target time window corresponding to the current time; based on the number of clients, constructing multiple prediction units, and initializing the weight of each prediction unit; obtaining the quota prediction value of each prediction unit in the target time window, and based on the weight of each prediction unit, determining the quota prediction value within the target time window.

[0109] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: based on the client, constructing a quota identification authentication request, wherein the quota identification authentication request at least includes a quota prediction value within a target time window.

[0110] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on a non-real-time asynchronous communication mechanism, using the client to send a constructed quota identification authentication request to the metadata server; when the metadata server receives the quota identification authentication request, it constructs quota identification authentication information and returns the quota identification authentication information to the client.

[0111] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the target time window, determining the expiration time of the quota identification authentication information; based on the number of quota identification authentication requests received by the metadata server, determining the incremental sequence number; based on the quota prediction value, determining the quota capacity; based on the application time and expiration time of the quota identification authentication information, the quota capacity and the incremental sequence number, generating a valid credential composed of an encrypted byte sequence, and defining the valid credential as the quota identification authentication information.

[0112] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining quota identification authentication information; and generating the write request based on the quota identification authentication information.

[0113] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the quota identification authentication information of the write request is detected to generate a detection result; in response to detecting that the detection result meets the preset standard, storage resources are allocated according to the quota capacity determined based on the quota prediction value, and returned to the client; in response to detecting that the detection result does not meet the preset standard, a prompt message is returned to the user end.

[0114] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sending a quota identification authentication request based on the first interface, responding to the quota identification authentication request based on the second interface, and generating quota identification authentication information; detecting the quota identification authentication information based on the third interface and generating a detection result; in response to detecting that the detection result meets a preset standard, providing the allocated storage resources to the client based on the fourth interface, and maintaining the allocated storage resources based on the fifth interface.

[0115] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining the real-time storage resource usage within the target time window; comparing the real-time storage resource usage within the target time window with the resource amount of the allocated storage resources to obtain a comparison result; and adjusting the weight of each prediction unit based on the comparison result.

[0116] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the difference between the real-time storage resource usage and the allocated storage resource usage; comparing the difference with a preset threshold; in response to detecting that the difference is greater than the preset threshold, obtaining a first comparison result; in response to detecting that the difference is less than or equal to the preset threshold, obtaining a second comparison result.

[0117] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to detecting that the comparison result is the first comparison result, based on the first weight adjustment algorithm, the weight of each prediction unit is calculated and determined; according to the weight of each prediction unit determined by the calculation, the weight of each prediction unit is adjusted.

[0118] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to detecting that the comparison result is the second comparison result, based on the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined; according to the weight of each prediction unit determined by the calculation, the weight of each prediction unit is adjusted.

[0119] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining a ratio of a prediction value of the i-th prediction unit in the t+1 prediction test to the sum of the prediction values ​​of all prediction units in the t+1 prediction test, the calculation formula of which is:

[0120] ;in, Represents the ratio, Indicates The predicted value of the prediction test, Indicates the number of prediction units.

[0121] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: based on the ratio and the first weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the first weight adjustment algorithm includes:

[0122] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0123] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: calculating the absolute value of the difference between the actual value and the predicted value to obtain a deviation value Based on the deviation value , calculate and determine the ratio corresponding to the deviation value, and the calculation formula is:

[0124] ;in, Indicates the ratio corresponding to the deviation value, Indicates the deviation value, Indicates the number of prediction units.

[0125] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: based on the ratio corresponding to the deviation value and the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the second weight adjustment algorithm includes:

[0126] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0127] In one embodiment, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the following steps are implemented: S1: obtaining a time series identifier corresponding to a client write request, and determining a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism; S2: based on the quota prediction value within the target time window, using the client to obtain quota identifier authentication information from the metadata server; S3: based on the quota identifier authentication information, generating a write request, and using the client to send the write request to a target storage node; S4: the target storage node allocates storage resources according to the quota identifier authentication information corresponding to the write request, and returns the storage resources to the client.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to detecting that a client issues a write request, identifying the write request based on the issuance time of the write request; and generating a time series identifier based on the time identifiers of multiple write requests within a target time period.

[0129] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: based on the time series identifier, determining the target time window corresponding to the current time; based on the number of clients, constructing multiple prediction units, and initializing the weight of each prediction unit; obtaining the quota prediction value of each prediction unit in the target time window, and based on the weight of each prediction unit, determining the quota prediction value within the target time window.

[0130] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: based on the client, constructing a quota identification authentication request, wherein the quota identification authentication request at least includes a quota prediction value within a target time window.

[0131] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on a non-real-time asynchronous communication mechanism, using the client to send a constructed quota identification authentication request to the metadata server; when the metadata server receives the quota identification authentication request, it constructs quota identification authentication information and returns the quota identification authentication information to the client.

[0132] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the target time window, determining the expiration time of the quota identification authentication information; based on the number of quota identification authentication requests received by the metadata server, determining the incremental sequence number; based on the quota prediction value, determining the quota capacity; based on the application time and expiration time of the quota identification authentication information, the quota capacity and the incremental sequence number, generating a valid credential composed of an encrypted byte sequence, and defining the valid credential as the quota identification authentication information.

[0133] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining quota identification authentication information; and generating the write request based on the quota identification authentication information.

[0134] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the quota identification authentication information of the write request is detected to generate a detection result; in response to detecting that the detection result meets the preset standard, storage resources are allocated according to the quota capacity determined based on the quota prediction value, and returned to the client; in response to detecting that the detection result does not meet the preset standard, a prompt message is returned to the user end.

[0135] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sending a quota identification authentication request based on the first interface, responding to the quota identification authentication request based on the second interface, and generating quota identification authentication information; detecting the quota identification authentication information based on the third interface and generating a detection result; in response to detecting that the detection result meets a preset standard, providing the allocated storage resources to the client based on the fourth interface, and maintaining the allocated storage resources based on the fifth interface.

[0136] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: obtaining the real-time storage resource usage within the target time window; comparing the real-time storage resource usage within the target time window with the resource amount of the allocated storage resources to obtain a comparison result; and adjusting the weight of each prediction unit based on the comparison result.

[0137] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the difference between the real-time storage resource usage and the allocated storage resource usage; comparing the difference with a preset threshold; in response to detecting that the difference is greater than the preset threshold, obtaining a first comparison result; in response to detecting that the difference is less than or equal to the preset threshold, obtaining a second comparison result.

[0138] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to detecting that the comparison result is the first comparison result, based on the first weight adjustment algorithm, the weight of each prediction unit is calculated and determined; according to the weight of each prediction unit determined by the calculation, the weight of each prediction unit is adjusted.

[0139] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in response to detecting that the comparison result is the second comparison result, based on the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined; according to the weight of each prediction unit determined by the calculation, the weight of each prediction unit is adjusted.

[0140] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining a ratio of a prediction value of the i-th prediction unit in the t+1 prediction test to the sum of the prediction values ​​of all prediction units in the t+1 prediction test, the calculation formula of which is:

[0141] ;in, Represents the ratio, Indicates The predicted value of the prediction test, Indicates the number of prediction units.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: based on the ratio and the first weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the first weight adjustment algorithm includes:

[0143] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0144] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: calculating the absolute value of the difference between the actual value and the predicted value to obtain a deviation value Based on the deviation value , calculate and determine the ratio corresponding to the deviation value, and the calculation formula is: ;in, Indicates the ratio corresponding to the deviation value, Indicates the deviation value, Indicates the number of prediction units.

[0145] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: based on the ratio corresponding to the deviation value and the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the second weight adjustment algorithm includes:

[0146] ;in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

[0147] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A quota management method, characterized in that: Applied to a distributed storage system, the distributed storage system includes at least a client, a metadata server, and multiple storage nodes, and the quota management method includes: Obtain a time series identifier corresponding to a client write request, and determine a quota prediction value within a target time window based on the time series identifier and a quota prediction mechanism, wherein the time series identifier is used to describe the time identifiers of multiple write requests within a target time period, and the quota prediction mechanism is used to determine a quota prediction value within the target time window based on the weight of each prediction unit; Based on the quota prediction value within the target time window, using the client to obtain quota identification authentication information from the metadata server, the quota identification authentication information is a valid credential composed of an encrypted byte sequence generated based on application time, expiration time, quota capacity and an incremental sequence number; Generate a write request based on the quota identification authentication information, and send the write request to a target storage node using the client; The target storage node allocates storage resources according to the quota identification authentication information corresponding to the write request, and returns the information to the client; After returning the allocated storage resources to the client, the method further comprises: Obtaining a difference between the real-time storage resource usage within the target time window and the resource amount of the allocated storage resource; In response to detecting that the difference is greater than a preset threshold, calculating and determining the weight of each prediction unit based on a first weight adjustment algorithm; According to the weight of each prediction unit determined by calculation, the weight of each prediction unit is adjusted.

2. The quota management method according to claim 1, characterized in that: The method for generating the time series identifier includes: In response to detecting that the client issues a write request, identifying the write request based on the issuance time of the write request; A time series identifier is generated based on the time identifiers of multiple write requests within the target time period.

3. The quota management method according to claim 1, characterized in that: Based on the time series identifier and the quota prediction mechanism, determining the quota prediction value within the target time window includes: Based on the time series identifier, determine the target time window corresponding to the current time; Based on the number of clients, build multiple prediction units and initialize the weight of each prediction unit; The quota prediction value of each prediction unit in the target time window is obtained, and the quota prediction value in the target time window is determined based on the weight of each prediction unit.

4. The quota management method according to claim 1, characterized in that: Before using the client to obtain quota identification authentication information from the metadata server, the method further includes: Based on the client, a quota identification authentication request is constructed, wherein the quota identification authentication request at least includes a quota prediction value within a target time window.

5. The quota management method according to claim 1, characterized in that: Based on the quota prediction value within the target time window, obtaining the quota identification authentication information from the metadata server using the client includes: Based on a non-real-time asynchronous communication mechanism, using a client to send a constructed quota identification authentication request to the metadata server; When receiving the quota identification authentication request, the metadata server constructs quota identification authentication information and returns the quota identification authentication information to the client.

6. The quota management method according to claim 1, characterized in that: The method for constructing the quota identification authentication information includes: Based on the target time window, determine the expiration time of the quota identification authentication information; Determine an incrementing sequence number based on the number of quota identification authentication requests received by the metadata server; Determine the quota capacity based on the quota forecast value; A valid credential consisting of an encrypted byte sequence is generated according to the application time and expiration time of the quota identification authentication information, the quota capacity and the incremental sequence number, and the valid credential is defined as the quota identification authentication information.

7. The quota management method according to claim 1, characterized in that: The method for generating the write request includes: Get quota identification authentication information; Based on the quota identification authentication information, the write request is generated.

8. The quota management method according to claim 1, characterized in that: After the target storage node receives the write request, the method further includes: Detecting the quota identification authentication information of the write request and generating a detection result; In response to detecting that the detection result meets the preset standard, allocating storage resources according to the quota capacity determined based on the quota prediction value, and returning the storage resources to the client; In response to detecting that the detection result does not meet the preset standard, a prompt message is returned to the user terminal.

9. The quota management method according to claim 1, characterized in that: The distributed storage system further includes a plurality of interfaces, and the method further includes: Sending a quota identification authentication request based on the first interface, and responding to the quota identification authentication request based on the second interface to generate quota identification authentication information; Detecting the quota identification authentication information based on the third interface and generating a detection result; In response to detecting that the detection result meets the preset standard, the allocated storage resources are provided to the client based on the fourth interface, and the allocated storage resources are maintained based on the fifth interface.

10. The quota management method according to claim 1, characterized in that: After returning the allocated storage resources to the client, the method further comprises: Get the real-time storage resource usage within the target time window; Comparing the real-time storage resource usage within the target time window with the resource amount of the allocated storage resources to obtain a comparison result; Based on the comparison result, the weight of each prediction unit is adjusted.

11. The quota management method according to claim 10, characterized in that: Comparing the real-time storage resource usage within the target time window with the resource amount of the allocated storage resources, and obtaining a comparison result includes: Obtaining a difference between the real-time storage resource usage and the allocated storage resource usage; comparing the difference with a preset threshold; In response to detecting that the difference is greater than a preset threshold, obtaining a first comparison result; In response to detecting that the difference is less than or equal to a preset threshold, a second comparison result is obtained.

12. The quota management method according to claim 11, characterized in that: Based on the comparison result, adjusting the weight of each prediction unit includes: In response to detecting that the comparison result is a second comparison result, calculating and determining the weight of each prediction unit based on a second weight adjustment algorithm; According to the weight of each prediction unit determined by calculation, the weight of each prediction unit is adjusted.

13. The quota management method according to claim 1, characterized in that: Before calculating and determining the weight of each prediction unit based on the first weight adjustment algorithm, the method further includes: Get the ratio of the predicted value of the i-th prediction unit in the t+1 prediction test to the sum of the predicted values ​​of all prediction units in the t+1 prediction test. The calculation formula is: ； in, Represents the ratio, Indicates The predicted value of the prediction test, Indicates the number of prediction units.

14. The quota management method according to claim 13, characterized in that: Based on the first weight adjustment algorithm, calculating and determining the weight of each prediction unit includes: Based on the ratio and the first weight adjustment algorithm, a weight of each prediction unit is calculated and determined, and the first weight adjustment algorithm includes: ； in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

15. The quota management method according to claim 12, characterized in that: Before calculating and determining the weight of each prediction unit based on the second weight adjustment algorithm, the method further includes: Calculate the absolute value of the difference between the actual value and the predicted value to get the deviation value ; Based on the deviation value , calculate and determine the ratio corresponding to the deviation value, and the calculation formula is: ； in, Indicates the ratio corresponding to the deviation value, Indicates the deviation value, Indicates the number of prediction units.

16. The quota management method according to claim 15, characterized in that: Based on the second weight adjustment algorithm, calculating and determining the weight of each prediction unit includes: Based on the ratio corresponding to the deviation value and the second weight adjustment algorithm, the weight of each prediction unit is calculated and determined, and the second weight adjustment algorithm includes: ； in, Indicates the step size of the optimal solution of the gradient loss function in each iteration, represents a linearly uniformly distributed weight vector, represents the weight of the prediction unit, Represents the normalization factor.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 16 is implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

19. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images