Quota calibration method and device, storage medium and electronic equipment

Abstract

The invention discloses a quota calibration method and device, a storage medium and electronic equipment, and relates to the technical field of computers, the quota calibration method comprises the steps that a quota calibration request is received, and the quota calibration request is used for requesting to calibrate an initial quota of a target object in a distributed storage system; in response to the quota calibration request, searching target metadata of a plurality of sub-objects included in the target object from a metadata server of the distributed storage system; detecting a target storage capacity required by the target object in the distributed storage system according to the target metadata; calibrating the initial quota according to the target storage capacity to obtain a target quota; and configuring the quota of the target object in the distributed storage system as the target quota. The technical problem that the utilization rate of the storage resources of the distributed storage system is low in the prior art is solved, and the technical effect of improving the utilization rate of the storage resources of the distributed storage system is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a quota calibration method and device, a storage medium, and an electronic device. Background Art

[0002] With the explosive growth of data volumes, distributed storage systems are widely used in various large-scale data storage scenarios due to their advantages such as high scalability, high availability, and high performance. In distributed storage systems, quota limits are usually set for different objects to rationally manage storage resources. This prevents certain objects from excessively occupying storage space, affecting the stable operation of the entire system and the normal use of other objects. Currently, such quota limits are often fixed matches set based on experience. However, in actual applications, due to the continuous writing, updating, and deletion of data, the actual space used by objects in the distributed storage system changes, resulting in a large deviation between the actual space used by objects and the quota limit. This can cause insufficient storage space allocated to objects in the distributed storage system, or waste of storage resources due to excessive quotas. Summary of the Invention

[0003] The present application provides a quota calibration method and device, a storage medium and an electronic device to at least solve the technical problem of low storage resource utilization of distributed storage systems in related technologies.

[0004] The present application provides a quota calibration method, comprising: receiving a quota calibration request, wherein the quota calibration request is used to request calibration of an initial quota of a target object in a distributed storage system; in response to the quota calibration request, searching for target metadata of multiple sub-objects included in the target object from a metadata server of the distributed storage system, wherein the target metadata is used to indicate the data access status of the corresponding sub-objects to the distributed storage system within a reference time period before a current moment; detecting a target storage capacity required by the target object in the distributed storage system based on the target metadata; calibrating the initial quota based on the target storage capacity to obtain a target quota; and configuring the quota of the target object in the distributed storage system to be the target quota.

[0005] The present application also provides a quota calibration device, comprising: a receiving module for receiving a quota calibration request, wherein the quota calibration request is used to request calibration of the initial quota of a target object in a distributed storage system; a search module for responding to the quota calibration request and searching for target metadata of multiple sub-objects included in the target object from a metadata server of the distributed storage system, wherein the target metadata is used to indicate the data access status of the corresponding sub-objects to the distributed storage system within a reference time period before a current moment; a detection module for detecting a target storage capacity required by the target object in the distributed storage system based on the target metadata; a calibration module for calibrating the initial quota based on the target storage capacity to obtain a target quota; and a configuration module for configuring the quota of the target object in the distributed storage system to be the target quota.

[0006] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned quota calibration methods when executing the computer program.

[0007] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned quota calibration methods are implemented.

[0008] The present application also provides a computer program product, comprising a computer program, which implements the steps of any of the above quota calibration methods when executed by a processor.

[0009] Through the present application, by searching the target metadata of multiple sub-objects of the target object from the metadata server of the distributed storage system, it is possible to know the data access status of the sub-objects to the distributed storage system within the reference time period before the current moment, and then the target storage capacity required by the target object in the distributed storage system can be detected based on the target metadata, so as to detect the storage capacity actually required by the target object in the distributed storage system based on the data access status of the sub-objects included in the target object to the distributed storage system within the reference time period, and then dynamically calibrate the initial quota of the target object in the distributed storage system according to the storage capacity actually required to be occupied by the target object, thereby avoiding excessive occupation or waste of distributed storage resources due to inaccurate quota configured for the object. Therefore, it can solve the technical problem of low storage resource utilization of the distributed storage system in the related technology, and achieve the technical effect of improving the utilization of storage resources of the distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 This is a hardware structure block diagram of the quota calibration method according to an embodiment of the present application;

[0012] Figure 2 is a flow chart of a quota calibration method according to an embodiment of the present application;

[0013] Figure 3 This is an optional distributed storage system quota calibration flow chart according to an embodiment of the present application;

[0014] Figure 4 This is a structural block diagram of a quota calibration device according to an embodiment of the present application. DETAILED DESCRIPTION

[0015] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0017] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the quota calibration method depends, the specific application environment architecture or specific hardware architecture is described herein.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1This is a hardware structure diagram of the quota calibration method of the embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown) a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU or a programmable logic device FPGA and other processing devices) and a memory 104 for storing data, wherein the above-mentioned server device may also include a transmission device 106 for communication functions and an input and output device 108. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the startup method of the operating system in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implementing the above-mentioned method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories can be connected to the server device via a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0021] The transmission device 106 is used to receive or send data via a network. A specific example of the aforementioned network may include a wireless network provided by a communication provider of the server device. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] An embodiment of the present application provides a quota calibration method, and the method is described in detail in conjunction with the execution flow of the quota calibration method.

[0023] The following is an explanation of the professional terms that appear in this application:

[0024] Quota: In storage systems, quotas are a resource management mechanism used to limit the amount of storage system resources, such as storage space and file count, that can be occupied by users, user groups, or specific directories. Setting quotas allows for the proper allocation and management of storage resources, preventing certain users or applications from excessively occupying resources and impacting the normal use of other users or applications, thereby ensuring fairness and stability in the storage system.

[0025] Distributed Storage System: A distributed storage system is a storage architecture that stores data in a distributed manner across multiple independent storage devices, which work together over a network. These storage devices are typically distributed across different locations or network nodes. The system uses specific distributed storage protocols and algorithms (such as data redundancy strategies and load balancing algorithms) to achieve data consistency, high availability, and high performance. Distributed storage systems offer greater scalability, reliability, and fault tolerance than traditional centralized storage systems, making them suitable for scenarios such as large-scale data storage and high-performance computing.

[0026] Quota Hard Threshold: Quota hard threshold refers to the strict upper limit set for storage resources (such as directories (objects), users, etc.) in quota management.

[0027] Quota Statistics of Used Amount: The used amount in quota management refers to the statistical value of the storage space or resource amount currently actually occupied by the system for storage resources (such as directories, users, etc.).

[0028] Ceph MDS (Ceph Metadata Server): Ceph MDS is the metadata server component in the Ceph distributed storage system.

[0029] Ceph MON (Ceph Monitor): Ceph MON is the monitoring component of the Ceph distributed storage system and one of the core management components of the Ceph cluster. The main responsibilities of Ceph MON include:

[0030] FUSE (Filesystem in Userspace): USE is a mechanism and technology that allows non-privileged users to create and operate file systems in user space.

[0031] In this embodiment, a quota calibration method is provided. Figure 2 is a flow chart of a quota calibration method according to an embodiment of the present application, such as Figure 2As shown, the method includes the following steps:

[0032] Step S202: receiving a quota calibration request, wherein the quota calibration request is used to request calibration of an initial quota of a target object in the distributed storage system;

[0033] Step S204: In response to the quota calibration request, searching the metadata server of the distributed storage system for target metadata of the plurality of sub-objects included in the target object, wherein the target metadata is used to indicate data access status of the corresponding sub-objects to the distributed storage system within a reference time period before the current moment;

[0034] Step S206: detecting the target storage capacity required by the target object in the distributed storage system according to the target metadata;

[0035] Step S208, calibrating the initial quota according to the target storage capacity to obtain a target quota;

[0036] Step S210: configuring the quota of the target object in the distributed storage system as the target quota.

[0037] Through the above steps, by searching the target metadata of multiple sub-objects of the target object from the metadata server of the distributed storage system, it is possible to know the data access status of the sub-objects to the distributed storage system within the reference time period before the current moment, and then the target storage capacity required by the target object in the distributed storage system can be detected based on the target metadata, so as to detect the storage capacity actually required by the target object in the distributed storage system based on the data access status of the sub-objects included in the target object to the distributed storage system within the reference time period, and then dynamically calibrate the initial quota of the target object in the distributed storage system according to the storage capacity actually required to be occupied by the target object, thereby avoiding excessive occupation or waste of distributed storage resources due to inaccurate quota configured for the object. Therefore, it can solve the technical problem of low storage resource utilization of the distributed storage system in the related technology, and achieve the technical effect of improving the utilization of storage resources of the distributed storage system.

[0038] The quota calibration method provided in the embodiments of the present application is applied to a device that has the function of configuring quotas for a distributed storage system. The device may be, but is not limited to, a quota configuration device configured for the distributed storage system. The quota configuration device is connected to a storage device included in the distributed storage system, thereby dynamically calibrating the quota of an object in the distributed storage system. Alternatively, the quota configuration device may be any storage device included in the distributed storage system. By running the quota calibration method on any storage device included in the distributed storage system, dynamic calibration of the quota of an object in the distributed storage system is achieved by running the quota calibration method.

[0039] In the embodiment provided in step S202, the quota calibration request may be, but is not limited to, generated when it is determined that the difference between the storage space capacity occupied by the target object in the distributed storage system and the storage space capacity indicated by the initial quota is greater than or equal to a target threshold. For example, an access request of the target object is received, wherein the access request is used to indicate that the target object requests to perform a data access operation on the distributed storage system; the target data volume of the data accessed by the data access operation is detected, and when the target data volume is greater than or equal to the target data volume threshold, it can be determined that there is a sudden change in the storage space occupied by the target object on the distributed storage system, that is, it is determined that the difference between the storage space capacity occupied by the target object in the distributed storage system and the storage space capacity indicated by the initial quota is greater than or equal to the target threshold.

[0040] In the embodiment provided in step S204, the multiple sub-objects included in the target object have completely independent access to the distributed storage system, and the total amount of storage resources allowed to be occupied by all sub-objects in the distributed storage system is less than or equal to the initial quota.

[0041] In the embodiment provided in step S206, the metadata server is used to store metadata for the distributed storage system. Each time an object performs a data access operation on the distributed storage system, metadata corresponding to the data access operation is generated on the metadata server. In this embodiment, the distributed storage can classify and store metadata based on the objects. That is, each time a quota is configured for an object in the distributed storage system, a metadata set corresponding to the object is created in the metadata server, and the metadata corresponding to the object is stored in the metadata set. The target metadata corresponding to the sub-object can then be found by searching the metadata server for the metadata set corresponding to the sub-object. Alternatively, the metadata can also carry the object identifier of the object to which it belongs, and the target metadata corresponding to the sub-object can then be found in the metadata server by searching the object identifier. This solution does not limit this.

[0042] In the embodiment provided in step S208, the target metadata is used to record the operation information of the data access operation performed by the corresponding sub-object in the distributed storage system, that is, each time the sub-object performs a data access operation, a metadata will be generated in the metadata server of the distributed storage system, and the metadata is used to represent the operation type (read data, write data), data access volume, execution time of the data access operation, execution location of the data access operation (the memory where the accessed data is located), and other information of the current data access operation. Then, by sorting the metadata of each target sub-object in chronological order, and summarizing the data accessed by each sub-object according to the operation type and data access volume of the access operation indicated by the metadata, the amount of data stored in the distributed storage system of each sub-object is obtained, that is, the storage capacity occupied by each sub-object in the distributed memory, and then by summarizing the storage capacity occupied by each sub-object, the target storage capacity of the target object is obtained.

[0043] Optionally, in an embodiment of the present application, since the multiple sub-objects included in the target object access the distributed storage system completely independently, that is, the operation type of the access operation performed by each sub-object on the distributed storage system is not fixed, the amount of data accessed is not fixed, and the storage location of the accessed data on the distributed storage system is not fixed, therefore, when statistics are performed on the capacity of the storage space occupied by the target object in the distributed storage system, it is necessary to first count the occupation of each memory on the distributed storage system by the sub-objects, and then count the occupation of the distributed memory by all sub-objects included in the target object, so as to obtain the actual occupation of the target object on the distributed storage system. Since this process requires the detection of the data storage status of each memory on the distributed storage system, the statistical delay is relatively large. Therefore, it is possible to predict the target storage capacity required by the target object on the distributed storage system in a candidate time period after the current moment based on the occupation information of the target object on the distributed storage system in a reference time period before the current moment, thereby achieving advanced prediction of the target storage capacity and dynamic calibration of the initial quota, so that the quota of the target object is more closely matched with the actual storage space occupation demand of the target object on the distributed storage system. In an embodiment of the present application, a method for monitoring target storage capacity based on target metadata may include, but is not limited to: converting the reference storage capacity required by the target object at the reference moment corresponding to the target metadata in the reference time period on the distributed storage system according to the target metadata; sorting the reference storage capacity of the target object at each reference moment in a plurality of reference moments according to the chronological order of the components to obtain a storage capacity sequence, wherein the reference time period includes a plurality of reference moments, the reference storage capacity is used to indicate the storage space occupied by the target object in the distributed storage system at the corresponding reference moment, and the storage occupancy information includes the reference storage capacity; using the storage capacity sequence to construct a target capacity curve for the target object, wherein the target capacity curve is used to indicate the relationship between the storage capacity demand of the target storage for the distributed storage system and time; inputting the target capacity curve at the target moment after the current moment of the capacity demand to be predicted, thereby obtaining the target storage capacity required by the target object on the distributed storage system at the target moment output by the target capacity curve. By the above method, the storage capacity occupation demand of the target object at a future moment is predicted based on the storage capacity occupation of the distributed storage system by the target object in the time period before the current moment, thereby realizing dynamic calibration of the quota.

[0044] In the embodiment provided in step S208, the method for calibrating the initial quota according to the target storage capacity may be: obtaining the first storage capacity occupied by the target object on the distributed storage system at each candidate moment in the candidate event segment before the current moment; calculating the capacity mean of the storage capacity required by the target object in the candidate time period according to the first storage capacity; calculating the capacity difference between the capacity mean and the target storage capacity; when the capacity difference is less than or equal to the set threshold, it can be determined that the current demand for storage capacity of the target object is a normal capacity demand, and the target storage capacity is used as the target quota; when the capacity difference is greater than the set threshold, it can be determined that the current demand for storage capacity of the target object is an abnormal capacity demand, and the initial quota is not calibrated.

[0045] As an optional implementation manner, detecting the target storage capacity required by the target object in the distributed storage system according to the target metadata includes:

[0046] Detecting, based on the target metadata, storage occupancy information of the target object on the distributed storage system during a reference time period, wherein the storage occupancy information is used to indicate storage space occupied by the target object on the distributed storage system during the reference time period;

[0047] The target storage capacity required by the target object in the distributed storage system at the current moment is generated according to the storage occupancy information.

[0048] Optionally, in an embodiment of the present application, the target metadata is used to record operational information about data access operations performed by the corresponding sub-object in the distributed storage system. That is, each time a sub-object performs a data access operation, a piece of metadata is generated in the metadata server of the distributed storage system. Furthermore, by constructing a metadata sequence corresponding to each sub-object in chronological order based on the target metadata of each sub-object, and sequentially calculating and analyzing the metadata at each time point in the metadata sequence according to the chronological order in the metadata sequence, the amount of storage space occupied by the sub-object in the distributed storage system at each time point included in the reference time period is obtained. The storage occupancy information includes the amount of storage space occupied by each sub-object included in the target object in the distributed storage system at each time point included in the reference time period.

[0049] Optionally, in an embodiment of the present application, the target storage capacity can be obtained by predicting the occupancy demand at the current moment using the storage occupancy information of the target object within a reference time period, or it can also be obtained by converting the reference storage capacity occupied by the target object on the distributed storage system within the reference time period based on the storage occupancy information, and then determining the reference storage capacity as the target storage capacity currently required by the target object.

[0050] Through the above content, by detecting the occupancy information of the distributed storage system through the target metadata, and then generating the target storage capacity currently required by the target object on the distributed storage system based on the storage occupancy information, it is possible to predict the target object's current demand for the storage capacity of the distributed storage system based on the distributed storage system unit metadata information detection method, improve the detection efficiency of the target object's demand for the storage capacity of the distributed storage system, and avoid the problem of inaccurate detection of the target object's occupancy of the distributed storage system due to detection delay problems in related technologies.

[0051] As an optional implementation manner, generating the target storage capacity required by the target object in the distributed storage system at the current moment according to the storage occupancy information includes:

[0052] Sort the reference storage capacity of the target object at each reference moment in a plurality of reference moments in chronological order to obtain a storage capacity sequence, wherein the reference time period includes the plurality of reference moments, the reference storage capacity is used to indicate the storage space occupied by the target object at the corresponding reference moment in the distributed storage system, and the storage occupancy information includes the reference storage capacity;

[0053] Calculating the capacity differences between the reference storage capacities at adjacent reference moments in the storage capacity sequence to obtain corresponding capacity changes at the reference moments;

[0054] Building a capacity change model of the target object using the capacity change at each reference moment, wherein the capacity change model records a relationship between a change in the storage capacity demand of the target object for the distributed storage system and time;

[0055] Inputting the time information of the current moment into the capacity change model, obtaining the capacity change model outputting the target demand change of the target object at the current moment;

[0056] The target demand change is used to adjust the reference storage capacity at the last moment in the storage capacity sequence to obtain the target storage capacity.

[0057] Optionally, in an embodiment of the present application, the capacity change model may be, but is not limited to, obtained by training an initial model using the capacity change at each reference moment as a training sample.

[0058] Optionally, in an embodiment of the present application, the capacity change is used to indicate the capacity difference between a first reference storage capacity occupied by the target object on the distributed storage system at the current reference moment and a second reference storage capacity occupied by the target object at the reference moment between the current reference moment.

[0059] Optionally, in an embodiment of the present application, the target storage capacity is obtained by adding the target demand change to the reference storage capacity at the last moment in the storage capacity sequence.

[0060] Through the above content, by calculating the capacity change of the target object at each reference time and constructing a capacity change model, it is possible to accurately predict the change in capacity demand at future times through the capacity change model, and then accurately predict the actual demand at future times based on the demand change, thereby improving the accuracy and reliability of the prediction results.

[0061] As an optional implementation manner, detecting, according to the target metadata, storage occupancy information of the target object in the distributed storage system during a reference time period includes:

[0062] Filtering a plurality of reference metadata at each reference moment from the target metadata, wherein the reference metadata is used to indicate a data access operation performed by the corresponding sub-object on the distributed storage system at the reference moment;

[0063] Converting the reference metadata into the initial storage capacity occupied by the sub-object at the corresponding reference time in the distributed storage system;

[0064] The initial storage capacities of all the sub-objects included in the object at the reference time are summarized to obtain the target storage capacity occupied by the target object in the distributed storage system at the reference time, wherein the storage occupancy information includes the target storage capacity.

[0065] Optionally, in an embodiment of the present application, the reference metadata may include, but is not limited to, indicating the operation type (i.e., read operation or write operation) of the access operation performed on the hard sub-object on the distributed storage system at the corresponding moment, and the data access volume.

[0066] As an optional implementation manner, calibrating the initial quota according to the target storage capacity to obtain the target quota includes:

[0067] Matching the target storage capacity with the initial quota to obtain a target matching result;

[0068] generating quota calibration information for the initial quota according to the matching result;

[0069] The initial quota is calibrated according to the quota calibration information to obtain the target quota.

[0070] Optionally, in an embodiment of the present application, the method for generating quota calibration information for the initial quota based on the matching result may be: when the target matching result is the target capacity difference between the target storage capacity and the initial quota, determining the target quota calibration amount corresponding to the target capacity difference amount from the capacity difference amounts and quota calibration amounts with a corresponding relationship, wherein the quota calibration information includes the target quota calibration amount.

[0071] Through the above content, dynamic calibration of the initial quota is achieved according to the matching relationship between the target storage capacity and the initial quota, thereby improving the matching relationship between the target quota and the capacity demand of the target object for the distributed storage system.

[0072] As an optional implementation manner, generating quota calibration information for the initial quota according to the matching result includes:

[0073] If the matching result indicates that the storage capacity indicated by the initial quota is greater than the target storage capacity, calculating a target capacity difference between the storage capacity indicated by the initial quota and the target storage capacity; determining a quota calibration amount corresponding to the target capacity difference as a target quota calibration amount, wherein the quota calibration information includes calibrating the initial quota using the target quota calibration amount;

[0074] If the matching result indicates that the storage capacity indicated by the initial quota is equal to the target storage capacity, determining the quota calibration information to not perform a calibration operation on the initial quota;

[0075] When the matching result indicates that the storage capacity indicated by the initial quota is smaller than the target storage capacity, and the target capacity difference is greater than or equal to a set threshold, the quota calibration information is determined to be not to perform a calibration operation on the initial quota.

[0076] As an optional implementation manner, searching the metadata server of the distributed storage system for target metadata of the multiple sub-objects included in the target object includes:

[0077] Obtaining the object identifier corresponding to the sub-object;

[0078] The target metadata corresponding to the object identifier is searched from the metadata log set of the metadata server.

[0079] As an optional implementation, the present application also provides a distributed storage system quota automatic calibration process and method, which can automatically identify the master node, comprehensively check the storage system status, and accurately calibrate the quota to ensure the stable operation of the storage system and the rational use of resources. Figure 3 This is an optional distributed storage system quota calibration flow chart according to an embodiment of the present application, such as Figure 3 As shown, it shall include at least the following contents:

[0080] 1. Determine the master node: First, determine whether the current node is the master monitor node. Only the master monitor node will continue with subsequent calibration operations to avoid problems caused by simultaneous calibration of multiple nodes. Obtain cluster status information and the master monitor node name by calling the Ceph cluster status get command, and compare it with the local node host name to determine whether it is the master monitor node.

[0081] 2. Check the cluster MDS status: Check the status of the metadata server (MDS) to ensure that the MDS is operating normally and can accurately provide directory metadata information, that is, the actual directory capacity, to provide a reliable basis for subsequent quota calibration. If the MDS status is abnormal, the calibration process is stopped and the abnormality log is recorded.

[0082] 3. Mount point check and creation: Check whether the specified FUSE mount point exists. If not, create it to ensure that subsequent operations can proceed smoothly.

[0083] 4. FUSE mount status check and processing: Perform a detailed check on the FUSE (Filesystem in Userspace) mount status, including whether the mount point exists and whether the FUSE process is running normally, and take appropriate measures based on the different statuses. If the FUSE is mounted but the process is not present, the mount point is unmounted; if the process is present but not mounted, the process is killed; if neither the process nor the mount point exists, the mount operation is performed directly.

[0084] 5. Obtain quota information and calibration: Ensure that FUSE is mounted properly. Obtain quota information for all directories in the cluster, including hard thresholds, soft thresholds, and used space. Then, compare the actual used space of the directory with the quota statistics. If the difference exceeds the set error threshold, calibrate the quota statistics by calling the relevant command and update the actual used space of the directory in the quota management system.

[0085] A distributed storage system quota automatic calibration process and method includes the following steps: determining the master mon node; checking the MDS status; checking and creating mount points; checking and processing the FUSE mount status; obtaining quota information and calibration. The process is as follows:

[0086] 1) Master Node Determination: First, determine whether the current node is the master monitor node. Only the master monitor node will continue with subsequent calibration operations to avoid issues caused by simultaneous calibration of multiple nodes. The ceph cluster status command is called to obtain cluster status information and the master monitor node name. This is then compared with the local node host name to determine whether the node is the master monitor node.

[0087] 2) Check the cluster MDS status: This check ensures that the MDS is functioning properly and accurately provides directory metadata, i.e., the actual directory capacity, to provide a reliable basis for subsequent quota calibration. If the MDS status is abnormal, the calibration process is stopped and the abnormality log is recorded.

[0088] 3) Mount point check and creation: Check whether the specified FUSE mount point exists. If not, create it to ensure that subsequent operations can proceed smoothly.

[0089] 4) FUSE Mount Status Check and Processing: Detailed checks are performed on the FUSE (Filesystem in Userspace) mount status, including whether the mount point exists and whether the FUSE process is running properly. Appropriate actions are taken based on the status. If the FUSE is mounted but the process is not present, the mount point is unmounted; if the process is present but not mounted, the process is killed; if neither the process nor the mount point exists, the mount operation is performed directly.

[0090] 5) Quota Information Acquisition and Calibration: While ensuring FUSE is mounted properly, obtain quota information for all directories in the cluster, including hard thresholds, soft thresholds, and used space. Then, compare the actual used space of the directory with the quota statistics. If the difference exceeds the set error threshold, calibrate the quota statistics by calling the relevant command and update the actual used space of the directory in the quota management system.

[0091] In the above embodiment, first, it is determined whether the current node is the master MON node. Only the master node will continue to perform subsequent quota calibration operations, avoiding resource contention and data inconsistency caused by simultaneous calibration of multiple nodes. Cluster MDS status check: Checks whether the status of the metadata server (MDS) in the cluster is normal, ensuring that the metadata information of the directory can be accurately obtained, providing a reliable basis for subsequent quota calibration. If the MDS status is abnormal, the calibration process is stopped and a log is recorded. USE mount point check and processing: Checks whether the FUSE mount point exists. If not, it is automatically created. Then, the FUSE mount status and process status are checked, and different processing is performed according to different combinations, such as unmounting the mounted FUSE, killing the running FUSE process, and forcibly unmounting the mounted FUSE but without a process, to ensure that the FUSE is in a normal state for subsequent operations. Quota information acquisition and calibration: Under the premise that the FUSE is mounted normally, the directory's quota information and metadata information are obtained. Based on this information, quota calibration operations are performed to ensure that the actual used space of the directory is consistent with the records in the quota management system, thereby achieving automatic and accurate calibration of the distributed storage system quota.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0093] The embodiment of the present application also provides a quota calibration device, Figure 4 This is a structural block diagram of a quota calibration device according to an embodiment of the present application, such as Figure 4 As shown, the device includes:

[0094] A receiving module, configured to receive a quota calibration request, wherein the quota calibration request is used to request calibration of an initial quota of a target object in the distributed storage system;

[0095] a search module, configured to, in response to the quota calibration request, search the metadata server of the distributed storage system for target metadata of a plurality of sub-objects included in the target object, wherein the target metadata is used to indicate data access status of the corresponding sub-objects to the distributed storage system within a reference time period before a current moment;

[0096] A detection module, configured to detect a target storage capacity required by the target object in the distributed storage system according to the target metadata;

[0097] a calibration module, configured to calibrate the initial quota according to the target storage capacity to obtain a target quota;

[0098] A configuration module is configured to configure the quota of the target object in the distributed storage system as the target quota.

[0099] Through the above device, by searching the target metadata of multiple sub-objects of the target object from the metadata server of the distributed storage system, it is possible to know the data access status of the sub-objects to the distributed storage system within the reference time period before the current moment, and then the target storage capacity required by the target object in the distributed storage system can be detected based on the target metadata, so as to detect the storage capacity actually required by the target object in the distributed storage system according to the data access status of the sub-objects included in the target object to the distributed storage system within the reference time period, and then dynamically calibrate the initial quota of the target object in the distributed storage system according to the storage capacity actually required to be occupied by the target object, thereby avoiding excessive occupation or waste of distributed storage resources due to inaccurate quota configured for the object. Therefore, it can solve the technical problem of low storage resource utilization of the distributed storage system in the related technology, and achieve the technical effect of improving the utilization of storage resources of the distributed storage system.

[0100] Optionally, the detection module includes:

[0101] a detection unit, configured to detect, based on the target metadata, storage occupancy information of the target object in the distributed storage system during a reference time period, wherein the storage occupancy information indicates storage space occupied by the target object in the distributed storage system during the reference time period;

[0102] The first generating unit is configured to generate the target storage capacity required by the target object in the distributed storage system at a current moment according to the storage occupancy information.

[0103] Optionally, the first generating unit is configured to:

[0104] Sort the reference storage capacity of the target object at each reference moment in a plurality of reference moments in chronological order to obtain a storage capacity sequence, wherein the reference time period includes the plurality of reference moments, the reference storage capacity is used to indicate the storage space occupied by the target object at the corresponding reference moment in the distributed storage system, and the storage occupancy information includes the reference storage capacity;

[0105] Calculating the capacity differences between the reference storage capacities at adjacent reference moments in the storage capacity sequence to obtain corresponding capacity changes at the reference moments;

[0106] Building a capacity change model of the target object using the capacity change at each reference moment, wherein the capacity change model records a relationship between a change in the storage capacity demand of the target object for the distributed storage system and time;

[0107] Inputting the time information of the current moment into the capacity change model, obtaining the capacity change model outputting the target demand change of the target object at the current moment;

[0108] The target demand change is used to adjust the reference storage capacity at the last moment in the storage capacity sequence to obtain the target storage capacity.

[0109] Optionally, the detection unit is used to:

[0110] Filtering a plurality of reference metadata at each reference moment from the target metadata, wherein the reference metadata is used to indicate a data access operation performed by the corresponding sub-object on the distributed storage system at the reference moment;

[0111] Converting the reference metadata into the initial storage capacity occupied by the sub-object at the corresponding reference time in the distributed storage system;

[0112] The initial storage capacities of all the sub-objects included in the object at the reference time are summarized to obtain the target storage capacity occupied by the target object in the distributed storage system at the reference time, wherein the storage occupancy information includes the target storage capacity.

[0113] Optionally, the calibration module includes:

[0114] a matching unit, configured to match the target storage capacity with the initial quota to obtain a target matching result;

[0115] a second generating unit, configured to generate quota calibration information for the initial quota according to the matching result;

[0116] A calibration unit is configured to calibrate the initial quota according to the quota calibration information to obtain the target quota.

[0117] Optionally, the second generating module is used to:

[0118] If the matching result indicates that the storage capacity indicated by the initial quota is greater than the target storage capacity, calculating a target capacity difference between the storage capacity indicated by the initial quota and the target storage capacity; determining a quota calibration amount corresponding to the target capacity difference as a target quota calibration amount, wherein the quota calibration information includes calibrating the initial quota using the target quota calibration amount;

[0119] If the matching result indicates that the storage capacity indicated by the initial quota is equal to the target storage capacity, determining the quota calibration information to not perform a calibration operation on the initial quota;

[0120] When the matching result indicates that the storage capacity indicated by the initial quota is smaller than the target storage capacity, and the target capacity difference is greater than or equal to a set threshold, the quota calibration information is determined to be not to perform a calibration operation on the initial quota.

[0121] Optionally, the search module includes:

[0122] An acquiring unit, configured to acquire an object identifier corresponding to the sub-object;

[0123] The searching unit is configured to search the metadata log set of the metadata server for the target metadata corresponding to the object identifier.

[0124] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned quota calibration method embodiments.

[0125] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned quota calibration method embodiments when running.

[0126] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0127] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned quota calibration method embodiments are implemented.

[0128] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned quota calibration method embodiments are implemented.

[0129] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0130] The above is a detailed introduction to a quota calibration method and device, storage medium and electronic device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A quota calibration method, characterized in that: include: receiving a quota calibration request, wherein the quota calibration request is used to request calibration of an initial quota of a target object in the distributed storage system; In response to the quota calibration request, searching, from a metadata server of the distributed storage system, target metadata of a plurality of sub-objects included in the target object, wherein the target metadata is used to indicate data access status of the corresponding sub-objects to the distributed storage system within a reference time period before a current moment; detecting, according to the target metadata, a target storage capacity required by the target object in the distributed storage system; Calibrate the initial quota according to the target storage capacity to obtain a target quota; The quota of the target object in the distributed storage system is configured as the target quota.

2. The method according to claim 1, characterized in that The detecting, according to the target metadata, the target storage capacity required by the target object in the distributed storage system includes: Detecting, based on the target metadata, storage occupancy information of the target object on the distributed storage system during a reference time period, wherein the storage occupancy information is used to indicate storage space occupied by the target object on the distributed storage system during the reference time period; The target storage capacity required by the target object in the distributed storage system at the current moment is generated according to the storage occupancy information.

3. The method according to claim 2, characterized in that Generating the target storage capacity required by the target object in the distributed storage system at the current moment according to the storage occupancy information includes: Sort the reference storage capacity of the target object at each reference moment in a plurality of reference moments in chronological order to obtain a storage capacity sequence, wherein the reference time period includes the plurality of reference moments, the reference storage capacity is used to indicate the storage space occupied by the target object at the corresponding reference moment in the distributed storage system, and the storage occupancy information includes the reference storage capacity; Calculating the capacity differences between the reference storage capacities at adjacent reference moments in the storage capacity sequence to obtain corresponding capacity changes at the reference moments; Building a capacity change model of the target object using the capacity change at each reference moment, wherein the capacity change model records a relationship between a change in the storage capacity demand of the target object for the distributed storage system and time; Inputting the time information of the current moment into the capacity change model, obtaining the capacity change model outputting the target demand change of the target object at the current moment; The target demand change is used to adjust the reference storage capacity at the last moment in the storage capacity sequence to obtain the target storage capacity.

4. The method according to claim 2, characterized in that The detecting, according to the target metadata, storage occupancy information of the target object in the distributed storage system during a reference time period includes: Filtering a plurality of reference metadata at each reference moment from the target metadata, wherein the reference metadata is used to indicate a data access operation performed by the corresponding sub-object on the distributed storage system at the reference moment; Converting the reference metadata into the initial storage capacity occupied by the sub-object at the corresponding reference time in the distributed storage system; The initial storage capacities of all the sub-objects included in the object at the reference time are summarized to obtain the target storage capacity occupied by the target object in the distributed storage system at the reference time, wherein the storage occupancy information includes the target storage capacity.

5. The method according to claim 1, wherein The step of calibrating the initial quota according to the target storage capacity to obtain the target quota includes: Matching the target storage capacity with the initial quota to obtain a target matching result; generating quota calibration information for the initial quota according to the matching result; The initial quota is calibrated according to the quota calibration information to obtain the target quota.

6. The method according to claim 5, characterized in that Generating quota calibration information for the initial quota according to the matching result includes: If the matching result indicates that the storage capacity indicated by the initial quota is greater than the target storage capacity, calculating a target capacity difference between the storage capacity indicated by the initial quota and the target storage capacity; determining a quota calibration amount corresponding to the target capacity difference as a target quota calibration amount, wherein the quota calibration information includes calibrating the initial quota using the target quota calibration amount; If the matching result indicates that the storage capacity indicated by the initial quota is equal to the target storage capacity, determining the quota calibration information to not perform a calibration operation on the initial quota; When the matching result indicates that the storage capacity indicated by the initial quota is smaller than the target storage capacity, and the target capacity difference is greater than or equal to a set threshold, the quota calibration information is determined to be not to perform a calibration operation on the initial quota.

7. The method according to claim 1, characterized in that The step of searching the metadata server of the distributed storage system for target metadata of the plurality of sub-objects included in the target object includes: Obtaining the object identifier corresponding to the sub-object; The target metadata corresponding to the object identifier is searched from the metadata log set of the metadata server.

8. A quota calibration device, characterized in that: include: A receiving module, configured to receive a quota calibration request, wherein the quota calibration request is used to request calibration of an initial quota of a target object in the distributed storage system; a search module, configured to, in response to the quota calibration request, search the metadata server of the distributed storage system for target metadata of a plurality of sub-objects included in the target object, wherein the target metadata is used to indicate data access status of the corresponding sub-objects to the distributed storage system within a reference time period before a current moment; A detection module, configured to detect a target storage capacity required by the target object in the distributed storage system according to the target metadata; a calibration module, configured to calibrate the initial quota according to the target storage capacity to obtain a target quota; A configuration module is configured to configure the quota of the target object in the distributed storage system as the target quota.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the quota calibration method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the quota calibration method according to any one of claims 1 to 7 are implemented.

Images