Capacity expansion method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses a capacity expansion method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining file attribute information of all files in a to-be-expanded virtual machine, and dividing all files into at least one file cluster based on the file attribute information; obtaining the space occupation amount of each file cluster, and determining the capacity expansion granularity interval of each file cluster based on the space occupation amount; obtaining a preset fitness evaluation index, and determining the optimal capacity expansion granularity of each file cluster in the capacity expansion granularity intervals corresponding to different file clusters based on the preset fitness evaluation index; and expanding the capacity of the file directory where each file cluster is located based on the optimal capacity expansion granularity of each file cluster. According to the invention, different capacity expansion processing is carried out for different file clusters, so that the pertinence of capacity expansion can be improved; by determining the capacity expansion granularity interval of each file cluster, the capacity expansion range can be reasonably planned, the capacity expansion granularity is prevented from being too large or too small, and the utilization rate of storage resources is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a capacity expansion method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of cloud computing and virtualization technologies, virtual machines, as the core carriers for resource isolation and flexible deployment, need to dynamically expand their storage capacity according to business load to meet the growing demand for file data.

[0003] In existing technologies, virtual machine capacity expansion often adopts a uniform granularity expansion scheme, that is, setting a fixed expansion range based on the current overall space occupancy of the virtual machine (such as expanding by a fixed ratio or a fixed capacity value), or directly manually configuring expansion parameters based on the administrator's experience to complete the batch expansion of the virtual machine file storage directory.

[0004] However, the existing solution has obvious technical flaws: it does not take into account the differences between different files within the virtual machine, which makes the expansion granularity lack specificity. This can easily lead to some file clusters needing to be expanded frequently due to excessively small expansion granularity, affecting the operating efficiency of the virtual machine; while some file clusters are wasted storage resources due to excessively large expansion granularity, reducing resource utilization. Summary of the Invention

[0005] This invention provides a capacity expansion method, apparatus, electronic device, storage medium, and program product to improve the targeting of capacity expansion and avoid storage resource utilization problems caused by excessively large or small expansion granularity.

[0006] In one aspect of this invention, a capacity expansion method is provided, comprising:

[0007] Obtain the file attribute information of all files in the virtual machine to be expanded, and divide all files into at least one file cluster based on the file attribute information;

[0008] Obtain the space usage of each file cluster, and determine the capacity expansion granularity range of each file cluster based on the space usage;

[0009] Obtain preset fitness evaluation indicators, and determine the optimal capacity expansion granularity for each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation indicators.

[0010] The capacity of the file directory containing each file cluster is expanded based on the optimal capacity expansion granularity for each file cluster.

[0011] In one aspect of this invention, a capacity expansion device is provided, comprising:

[0012] The file cluster partitioning module is used to obtain the file attribute information of all files in the virtual machine to be expanded, and to partition all files into at least one file cluster based on the file attribute information;

[0013] The interval determination module is used to obtain the space usage of each file cluster and the number of files contained in each file cluster, and to determine the capacity expansion granularity interval of each file cluster based on the space usage and the number of files.

[0014] The granularity determination module is used to obtain preset fitness evaluation indicators and determine the optimal capacity expansion granularity for each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation indicators.

[0015] The capacity expansion module is used to expand the capacity of the file directory containing each file cluster based on the optimal capacity expansion granularity for each file cluster.

[0016] In another aspect of the present invention, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] Memory that is communicatively connected to at least one processor;

[0019] The memory stores a computer program that can be executed by at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the capacity expansion method of any embodiment of the present invention.

[0020] In another aspect of the present invention, a computer-readable storage medium is provided, comprising: computer instructions, which enable a processor to execute the capacity expansion method of any embodiment of the present invention when executed.

[0021] This invention acquires file attribute information of all files in the virtual machine to be expanded, divides all files into at least one file cluster based on the file attribute information, acquires the space occupancy of each file cluster, determines the capacity expansion granularity range of each file cluster based on the space occupancy, acquires a preset fitness evaluation index, and determines the optimal capacity expansion granularity of each file cluster within the corresponding capacity expansion granularity range based on the preset fitness evaluation index. Based on the optimal capacity expansion granularity of each file cluster, the capacity of the file directory containing each file cluster is expanded. This invention improves the targeting of capacity expansion by implementing different capacity expansion processes for different file clusters; by determining the capacity expansion granularity range of each file cluster, the expansion range can be reasonably planned, avoiding expansion granularity that is too large or too small, thus improving the utilization rate of storage resources; by determining the optimal capacity expansion granularity based on the preset fitness evaluation index, it ensures that the virtual machine to be expanded reaches its optimal state after expansion, thereby improving the operating efficiency of the virtual machine to be expanded.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a capacity expansion method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of another capacity expansion method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a flowchart of another capacity expansion method provided in Embodiment 3 of the present invention;

[0027] Figure 4 This is a flowchart of another capacity expansion method provided according to Embodiment 4 of the present invention;

[0028] Figure 5 This is a structural diagram of a capacity expansion device provided in Embodiment 5 of the present invention;

[0029] Figure 6 This is a block diagram of an electronic device that performs a capacity expansion method according to Embodiment Six of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0032] Figure 1 This invention provides a flowchart of a capacity expansion method. This invention is applicable to scenarios where the existing resources of the virtual machine to be expanded cannot meet business needs. The method can be executed by a capacity expansion device, which can be implemented in hardware and / or software. This capacity expansion device can be configured in electronic devices such as a semiconductor testing server or a distributed computing cluster composed of multiple servers. Figure 1 As shown, the method includes:

[0033] S110. Obtain the file attribute information of all files in the virtual machine to be expanded, and divide all files into at least one file cluster based on the file attribute information.

[0034] Among them, virtual machines to be expanded refer to virtual computing nodes in a cloud computing virtualization environment that need to have their capacity expanded. Virtual machines to be expanded store a large number of files. When the existing resources of virtual machines to be expanded cannot meet business needs, it is necessary to expand the capacity of virtual machines to meet actual business needs.

[0035] File attribute information can be understood as a set of data describing file characteristics, used for file classification. Specifically, file attribute information can be divided into numerical attribute information and non-numerical attribute information. Numerical attribute information represents directly quantifiable characteristics of the file, such as file size, modification time, and access frequency. Non-numerical attribute information represents non-directly quantifiable characteristics of the file, such as file type and storage location.

[0036] A file cluster can be understood as a collection of files. By analyzing file attributes, files with similar characteristics are grouped into a file cluster.

[0037] Specifically, when the existing resources of the virtual machine to be expanded cannot meet business needs, all files stored in the virtual machine to be expanded can be traversed, and the file attribute information carried by each file can be extracted. Based on the similarity between the file attribute information of each file, all files in the virtual machine to be expanded can be divided into different file sets, which can be defined as file clusters. Files in the same file cluster have a high degree of similarity; files in different file clusters have a low degree of similarity.

[0038] S120. Obtain the space usage of each file cluster, and determine the capacity expansion granularity range of each file cluster based on the space usage.

[0039] The space occupancy refers to the amount of storage space occupied by a single file cluster in the virtual machine to be expanded, used to determine the capacity expansion granularity range of the corresponding file cluster. It is understandable that if the space occupancy of each file cluster is different, the determined capacity expansion granularity range of the file cluster will also be different; each file cluster has its own corresponding capacity expansion granularity range.

[0040] The capacity expansion granularity range refers to the granularity range of capacity expansion determined for each file cluster. It is used to define an effective granularity range in advance for determining the optimal capacity expansion granularity, and provides a granular boundary for subsequent screening of the optimal capacity expansion granularity.

[0041] Specifically, the total capacity of the directory containing each file cluster is obtained. This total capacity can be used as the space occupancy of each file cluster. An initial capacity expansion granularity can be preset. In a pre-created capacity test environment, the space occupancy of each file cluster can be simulated for expansion according to each preset initial capacity expansion granularity. The storage resource utilization rate of each file cluster after simulated expansion can be obtained. For each file cluster, a relationship curve between storage resource utilization rate and initial capacity expansion granularity can be plotted. The initial capacity expansion granularity range corresponding to the effective interval on the relationship curve can be used as the capacity expansion granularity interval. For example, the effective interval on the relationship curve can be a high utilization interval, a stable interval, or the intersection of two stable intervals, etc.

[0042] Furthermore, in this embodiment of the invention, when two or more file clusters reside in the same file directory, the space occupied by each file cluster in the same file directory can be queried by calling a preset query tool to obtain the space occupied by each file cluster. This embodiment of the invention does not impose any restrictions on the pre-query tool.

[0043] S130. Obtain the preset fitness evaluation index, and determine the optimal capacity expansion granularity of each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index.

[0044] Among them, the preset fitness evaluation indicators can be understood as a series of pre-set standards used to evaluate the quality of capacity expansion granularity. For example, preset fitness evaluation indicators may include indicators such as metadata management efficiency indicators, read and write time indicators, storage space utilization indicators, and fragmentation rate indicators.

[0045] The optimal capacity expansion granularity refers to the most suitable capacity expansion granularity for a file cluster selected within a certain range. The optimal capacity expansion granularity is the final execution standard for expanding the file directory containing the file cluster. In this embodiment of the invention, the optimal capacity expansion granularity determined based on a preset fitness evaluation index can be understood as follows: after expanding the capacity of the file cluster based on the optimal capacity expansion granularity, the file cluster achieves an optimal balance in terms of metadata management efficiency, read / write time, storage space utilization, and fragmentation rate.

[0046] Specifically, pre-defined fitness evaluation indicators are obtained to assess the quality of capacity expansion granularity. For example, the pre-defined fitness evaluation indicators may include indicators such as metadata management efficiency, read / write time, storage space utilization, and fragmentation rate. Each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters can be evaluated from dimensions such as metadata management efficiency, read / write time, storage space utilization, and fragmentation rate to determine the optimal capacity expansion granularity within the capacity expansion granularity range.

[0047] S140. Expand the capacity of the file directory where each file cluster is located based on the optimal capacity expansion granularity of each file cluster.

[0048] Specifically, based on the optimal capacity expansion granularity corresponding to each file cluster, the directory capacity of the file directory containing each file cluster is expanded. It can be understood that after expanding the capacity of the file directories containing each file cluster, the total directory capacity equals the sum of the space occupied by each file cluster and the optimal capacity expansion granularity.

[0049] This invention involves obtaining file attribute information for all files in the virtual machine to be expanded, dividing all files into at least one file cluster based on the file attribute information, obtaining the space occupancy of each file cluster, determining the capacity expansion granularity range for each file cluster based on the space occupancy, obtaining a preset fitness evaluation index, and determining the optimal capacity expansion granularity for each file cluster within the corresponding capacity expansion granularity range based on the preset fitness evaluation index, and expanding the capacity of the file directory containing each file cluster based on the optimal capacity expansion granularity. This invention improves the targeting of capacity expansion by implementing different capacity expansion processes for different file clusters; by determining the capacity expansion granularity range for each file cluster, the expansion range can be reasonably planned, avoiding excessively large or small expansion granularity, thus improving the utilization rate of storage resources; and by determining the optimal capacity expansion granularity based on the preset fitness evaluation index, the virtual machine to be expanded can be ensured to reach its optimal state after expansion, thereby improving the operating efficiency of the virtual machine to be expanded.

[0050] Furthermore, it can monitor the storage resource utilization of virtual machines to be expanded in real time. When the storage resource utilization exceeds the preset storage resource utilization threshold, the capacity of the file system where each file cluster is located is expanded.

[0051] Storage resource utilization rate refers to the proportion of the used storage space of the virtual machine to be expanded to the total storage space. The storage resource utilization rate of the virtual machine to be expanded is monitored in real time. When the storage resource utilization rate is greater than the preset storage resource utilization rate threshold, it indicates that the storage resources of the virtual machine to be expanded are tight. Expanding the capacity of the file directory where each file cluster is located cannot meet the storage demand. It is necessary to expand the capacity of the file system where each file cluster is located to meet the storage demand.

[0052] The preset storage resource utilization threshold can be understood as a pre-defined limit value used to indicate when to perform file system-level expansion. When the storage resource utilization of the virtual machine to be expanded exceeds the preset storage resource utilization threshold, file system-level expansion is performed, that is, the capacity of the file system where each file cluster resides is expanded.

[0053] Specifically, during the expansion process, the storage resource utilization of the virtual machine to be expanded can be monitored in real time, or it can be measured at fixed time intervals. Once the storage resource utilization of the virtual machine to be expanded is obtained, it can be compared with a preset storage resource utilization threshold. If the storage resource utilization of the virtual machine to be expanded is greater than the preset storage resource utilization threshold, it indicates that after expanding the capacity of the file directory where each file cluster is located, the existing resources of the virtual machine to be expanded still cannot meet the business needs. In this case, file system-level expansion is required, that is, expanding the capacity of the file system where each file cluster is located.

[0054] Example 2

[0055] Figure 2 This invention provides a flowchart of another capacity expansion method for Embodiment 2 of the present invention. This embodiment of the present invention is a refinement of the above embodiment, specifically refining the specific steps for determining the granularity range of capacity expansion.

[0056] like Figure 2 As shown, another capacity expansion method may include the following specific steps:

[0057] S210. Obtain the file attribute information of all files in the virtual machine to be expanded, and divide all files into at least one file cluster based on the file attribute information.

[0058] S220. Obtain the pre-configured initial capacity expansion granularity range for each file cluster; wherein the initial capacity expansion granularity range contains at least one initial capacity expansion granularity.

[0059] The initial capacity expansion granularity range refers to multiple pre-set capacity expansion granularity schemes for different file clusters. These schemes provide the initial granularity range for subsequent simulated capacity expansion in a pre-defined test environment. By testing and analyzing different initial capacity expansion granularities within this range, a more precise capacity expansion granularity range can be further determined. For example, the initial capacity expansion granularity range could be 100MB to 1GB, or 700MB to 1100MB, etc. It is understood that the capacity expansion granularity range is included within the initial capacity expansion granularity range.

[0060] The initial capacity expansion granularity refers to the specific capacity expansion value within the initial capacity expansion granularity range. The interval between each initial capacity expansion granularity can be 100MB. For example, if the initial capacity expansion granularity range is from 100MB to 1GB, then the initial capacity expansion granularity within the initial capacity expansion granularity range can be 100MB, 200MB, 300MB, etc., increasing in 100MB intervals until reaching 1GB. The interval between each initial capacity expansion granularity can also be 200MB. For example, if the initial capacity expansion granularity range is from 100MB to 3GB, then the initial capacity expansion granularity within the initial capacity expansion granularity range can be 100MB, 300MB, 500MB, etc., increasing in 200MB intervals until reaching 3GB.

[0061] Specifically, based on the historically stored data and / or business requirements in the file cluster, an initial capacity expansion granularity range can be pre-configured for each file cluster. The initial capacity expansion granularity ranges for each file cluster can be the same or different, but each file cluster's initial capacity expansion granularity range must include at least one initial capacity expansion granularity. The interval between the initial capacity expansion granularities within the initial capacity expansion granularity range can also be set according to the historically stored data and / or business requirements in the file cluster; this embodiment of the invention does not impose any limitations.

[0062] S230. In a preset test environment, the capacity of the file directory where each file cluster is located is simulated and expanded one by one according to different initial capacity expansion granularities within the initial capacity expansion granularity range, and the storage resource utilization rate of each file cluster after the simulated expansion is determined based on the space occupancy.

[0063] The preset test environment refers to a pre-built environment used to test capacity expansion operations. In the preset test environment, the capacity of the file directory where each file cluster is located can be simulated to expand according to different initial capacity expansion granularities without affecting the actual storage resources. Based on the simulation results of the simulated expansion in the preset test environment, a more accurate capacity expansion granularity range can be further determined within the initial capacity expansion granularity range.

[0064] Specifically, before actually expanding the capacity of the virtual machine to be expanded, a preset test environment can be built to simulate the virtual expansion of the virtual machine. In the preset test environment, for each file cluster, simulated expansion can be performed on the file directory where the file cluster is located, according to different initial capacity expansion granularities within its corresponding initial capacity expansion granularity range. After the simulated expansion, the used capacity and total capacity of the file directory where the file cluster is located can be counted to obtain the storage resource utilization rate of each file cluster under different initial capacity expansion granularities. Among them, the space occupied by the file cluster and the cumulative value of the initial capacity expansion granularity can be used as the used capacity after simulated expansion.

[0065] S240. For each file cluster, a multinomial regression algorithm is used to fit the storage resource utilization rate with the initial capacity expansion granularity to obtain the relationship curve between the storage resource utilization rate and the initial capacity expansion granularity for each file cluster.

[0066] Specifically, after simulating expansion for each file cluster, the storage resource utilization rate of each file cluster after simulated expansion according to different initial capacity expansion granularities can be obtained. In other words, for each file cluster, the initial capacity expansion granularity and the storage resource utilization rate are in one-to-one correspondence. A multinomial regression algorithm can be used to fit the curve of the initial capacity expansion granularity and the storage resource utilization rate corresponding to each file cluster, so as to obtain the relationship curve between the storage resource utilization rate and the initial capacity expansion granularity for each file cluster.

[0067] S250. In the relationship curve, extract the high utilization range where the storage resource utilization rate is greater than the preset storage resource utilization rate threshold, as well as the stable range of the relationship curve.

[0068] The high utilization range refers to the range on the relationship curve where the storage resource utilization rate is greater than the preset storage resource utilization rate threshold; the stable range refers to the range on the relationship curve where the storage resource utilization rate is relatively stable and fluctuates little. It can be understood that under the initial capacity expansion granularity corresponding to the stable range, the storage resource utilization rate of the virtual machine to be expanded is in a relatively stable state.

[0069] Specifically, a preset storage resource utilization threshold can be set in advance to determine whether the storage resource utilization efficiency meets the standard. The relationship curve can be traversed to extract the initial capacity expansion granularity range where the storage resource utilization is higher than the preset storage resource utilization threshold, which is taken as the high utilization range. A storage resource utilization fluctuation threshold can be set in advance, and the utilization difference corresponding to the continuous granularity in the relationship curve can be calculated. The initial capacity expansion granularity range where the utilization difference is less than the storage resource utilization fluctuation threshold can be extracted as the stable range.

[0070] Furthermore, in this embodiment of the invention, the relationship curve between storage resource utilization and initial capacity expansion granularity can be visualized, allowing staff to directly observe and obtain the high utilization range and the stable range.

[0071] S260. The initial capacity expansion granularity range corresponding to the intersection of the high utilization range and the stable range shall be taken as the capacity expansion granularity range.

[0072] The intersection interval refers to the overlapping part of the high utilization interval and the stable interval. The initial capacity expansion granularity range corresponding to the intersection interval can be used as the capacity expansion granularity interval.

[0073] Specifically, the boundaries of the high utilization interval and the stable interval can be compared to obtain the intersection interval of the high utilization interval and the stable interval. The initial capacity expansion granularity range corresponding to the intersection interval is used as the final capacity expansion granularity interval for each file cluster.

[0074] S270. Obtain the preset fitness evaluation index, and determine the optimal capacity expansion granularity for each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index.

[0075] S280. Expand the capacity of the file directory where each file cluster is located based on the optimal capacity expansion granularity of each file cluster.

[0076] In this embodiment of the invention, file attribute information of all files in the virtual machine to be expanded is obtained, and all files are divided into at least one file cluster based on the file attribute information. An initial capacity expansion granularity range is obtained for each file cluster. Each initial capacity expansion granularity range contains at least one initial capacity expansion granularity. In a preset test environment, the capacity of the file directory containing each file cluster is simulated for expansion according to different initial capacity expansion granularities within the initial capacity expansion granularity range. The storage resource utilization rate of each file cluster after simulated expansion is determined based on the space occupancy. For each file cluster, a multinomial regression algorithm is used to correlate the storage resource utilization rate with the initial capacity expansion granularity. Curve fitting is performed to obtain the relationship curve between the storage resource utilization rate and the initial capacity expansion granularity for each file cluster. From this curve, the high utilization rate interval (greater than a preset storage resource utilization threshold) and the stable interval are extracted. The initial capacity expansion granularity range corresponding to the intersection of the high utilization rate interval and the stable interval is taken as the capacity expansion granularity interval. A preset fitness evaluation index is obtained, and the optimal capacity expansion granularity for each file cluster is determined within the capacity expansion granularity interval corresponding to different file clusters based on the preset fitness evaluation index. The capacity of the file directory containing each file cluster is expanded based on the optimal capacity expansion granularity for each file cluster. This embodiment of the invention avoids the problem of long simulation cycles caused by full-range simulated expansion by presetting the initial expansion range, thus improving simulation efficiency and consequently improving the efficiency of capacity expansion. By using the intersection of the high utilization rate interval and the stable interval as the capacity expansion granularity interval, the one-sidedness of a single standard selection interval is avoided, improving the reliability of the capacity expansion granularity interval and thus enhancing the reliability of the capacity expansion method.

[0077] Furthermore, this embodiment of the invention further refines S230, specifically detailing the steps for determining the storage resource utilization rate of a file cluster, including:

[0078] S2301. Obtain the initial total directory capacity of the file directory where each file cluster is located.

[0079] The initial total directory capacity can be understood as the storage space of the file directory containing a single file cluster before the simulated expansion, reflecting the available space of the file directory containing a single file cluster before the simulated expansion. The initial total directory capacity can be used as the basic parameter for calculating the total expanded directory capacity. The accumulated value obtained by directly adding the initial capacity expansion granularity to the initial capacity expansion value can be used as the total expanded directory capacity after the simulated expansion of each file cluster.

[0080] Specifically, the space occupied by each file cluster can be used as the initial total directory capacity of the file directory where each file cluster is located.

[0081] S2302. Add each initial capacity expansion granularity to the initial total directory capacity to obtain the total directory expansion capacity corresponding to each initial capacity expansion granularity after the simulated expansion.

[0082] The total expanded directory capacity can be understood as the storage space size of the file directory containing a single file cluster after simulated expansion, reflecting the available space of the file directory after simulated expansion. Each file cluster has a unique initial total directory capacity and its own pre-configured initial capacity expansion granularity range. The file cluster can be expanded according to each initial capacity expansion granularity within the corresponding pre-configured range. This is achieved by adding the initial total directory capacity of the file cluster to each initial capacity expansion granularity, thus obtaining the total expanded directory capacity of the file cluster at each initial capacity expansion granularity.

[0083] Specifically, the capacity of the file directory containing each file cluster is simulated and expanded according to each initial capacity expansion granularity corresponding to each file cluster, so as to obtain the total expanded capacity of the directory corresponding to each initial capacity expansion granularity after the simulated expansion. The calculation steps of the total expanded capacity of the directory may include: adding the different initial capacity expansion granularities corresponding to each file cluster to the initial total capacity of the directory to obtain an accumulated value, which can be used as the total expanded capacity of the directory.

[0084] S2303. Obtain the simulation test period under the preset test environment, and the total new space occupied by each file cluster during the simulation test period.

[0085] The simulation test period can be understood as the duration of simulated expansion of file clusters in a preset test environment. For example, the simulation test period can be 7 days or 3 hours, etc. It is used to reflect the file growth cycle of the virtual machine to be expanded in the actual business scenario. It can ensure that the growth pattern of file clusters during the simulated expansion process is consistent with the real business, and avoid the deviation in the calculation of the total new space occupancy caused by the mismatch between the test period and the actual business cycle. It provides a time benchmark that conforms to the actual scenario for subsequent accurate evaluation of the resource adaptation capability of dynamically growing files at different initial capacity expansion granularities.

[0086] The total new space usage can be understood as the amount of storage space newly occupied by the file cluster during the simulation test period, which can reflect the growth of the space usage of the file cluster during the test.

[0087] Specifically, the daily growth rate of file cluster space usage can be obtained by analyzing the historical growth trend of file clusters. The product of the simulation test period, the growth rate, and the space usage can then be used as the total new space usage for each file cluster. In this embodiment of the invention, the future growth trend of file clusters is also considered, ensuring that sufficient space is reserved to cope with future growth during simulated expansion, avoiding the problem of insufficient capacity again in the short term after expansion due to not considering the growth trend.

[0088] S2304. Obtain the sum of the total newly added space usage and the total space usage, and use the ratio of the sum to the total capacity of the directory expansion as the storage resource utilization rate corresponding to each initial capacity expansion granularity.

[0089] Specifically, the total new space occupancy and the space occupancy corresponding to each initial capacity expansion granularity are added together to obtain the cumulative value of the total new space occupancy and the space occupancy. This cumulative value can be used as the numerator, and the total directory expansion capacity can be used as the numerator to obtain the ratio of the cumulative value to the total directory expansion capacity. This ratio can be used as the storage resource utilization rate corresponding to each initial capacity expansion granularity.

[0090] Example 3

[0091] Figure 3 This invention provides a flowchart of another capacity expansion method for Embodiment 3 of the present invention. This embodiment of the present invention is a refinement of the above embodiment, specifically refining the specific steps of how to divide file clusters and how to determine the optimal capacity expansion granularity.

[0092] like Figure 3 As shown, another capacity expansion method may include the following specific steps:

[0093] S310. Extract the numerical and non-numerical attribute information of each file in the virtual machine to be expanded from the file attribute information.

[0094] Numerical attribute information can be understood as the quantifiable characteristic data of a file. For example, numerical attribute information may include data such as file size, file modification time, and the access frequency of the file.

[0095] Non-numerical attribute information can be understood as characteristic data of a file that cannot be directly quantified. For example, non-numerical attribute information includes at least one of the following: file type and file storage location.

[0096] Specifically, obtain the file attribute information of each file in the virtual machine to be expanded, and extract the corresponding numerical and non-numerical attribute information for each file from the file attribute information of each file.

[0097] S320. Normalize the numerical attribute information to obtain the numerical feature vector of each file, and perform one-hot encoding on the non-numerical attribute information to obtain the binary feature vector of each file.

[0098] In this context, the numerical feature vector refers to the vector formed by arranging the normalized numerical attribute information after normalization. This vector, along with the binary feature vector, serves as input to the clustering algorithm for dividing file clusters. Normalization transforms numerical attribute information with different dimensions and value ranges into data of a uniform scale, facilitating subsequent data analysis and processing. For example, the normalized numerical attribute information can be input into a pre-defined K-means clustering algorithm model for file cluster division.

[0099] A binary feature vector is a vector formed by arranging the one-hot encoded non-numerical attribute information after it has been one-hot encoded. This vector, along with numerical feature vectors, is used to partition file clusters. One-hot encoding converts non-numerical attribute information into a binary form that is easily processed by the model. Each non-numerical attribute value corresponds to a unique binary vector. Concatenating these unique binary vectors yields the binary feature vector of the non-numerical attribute information.

[0100] Specifically, a file can correspond to at least one numerical attribute information. After normalizing the numerical attribute information, the normalized numerical attribute information can be arranged to obtain the numerical feature vector of each file. A file can also correspond to at least one non-numerical attribute information. The non-numerical attribute information can be converted into a binary form that is easy for the model to process through one-hot encoding. The binary form of the non-numerical attribute information after one-hot encoding is arranged to obtain the binary feature vector of each file.

[0101] S330. Call the preset K-means clustering algorithm model, concatenate the numerical feature vector and the binary feature vector and input them into the preset K-means clustering algorithm model to obtain at least one file cluster; wherein, the number of file clusters is the K value preset in the preset K-means clustering algorithm model.

[0102] The preset K-means clustering algorithm model can be understood as a machine learning algorithm model used to divide files into K clusters, where the value of K can be set according to the total number of files and business needs. The preset K-means clustering algorithm model groups similar files into the same cluster by calculating the Euclidean distance between the feature vectors of the files.

[0103] Specifically, the concatenated vectors of each file obtained by concatenating the numerical feature vectors and binary feature vectors can be input into a preset K-means clustering algorithm model. The preset K-means clustering algorithm model calculates the Euclidean distance between the concatenated vectors of the files and groups files with closer Euclidean distances, i.e., more similar files, into the same cluster. The preset K-means clustering algorithm model outputs at least K file clusters, where the value of K can be set according to the total number of files and business requirements.

[0104] S340. Obtain the space usage of each file cluster and determine the capacity expansion granularity range of each file cluster based on the space usage.

[0105] S350. Obtain the preset fitness evaluation index, and determine the fitness of each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index.

[0106] Among them, fitness can be understood as an indicator to reflect the quality of capacity expansion granularity. It is used to measure whether the capacity expansion granularity is most suitable for the expansion needs of the file cluster. For example, the higher the fitness, the more suitable the capacity expansion granularity is for the expansion needs of the corresponding file cluster.

[0107] Specifically, pre-defined fitness evaluation indicators are obtained. For example, the pre-defined fitness evaluation indicators may include indicators such as metadata management efficiency indicators, read / write time indicators, storage space utilization indicators, and fragmentation rate indicators. Each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters can be evaluated from dimensions such as metadata management efficiency indicators, read / write time indicators, storage space utilization indicators, and fragmentation rate indicators to obtain the fitness of each capacity expansion granularity within the capacity expansion granularity range.

[0108] For example, the steps for determining the fitness of each capacity expansion granularity may include: in a preset test environment, simulating capacity expansion of the file directory where each file cluster is located according to different capacity expansion granularities; obtaining the metadata management efficiency, read / write time, storage space utilization, and fragmentation rate of each file cluster after simulated expansion according to preset fitness evaluation indicators; and using the normalized sum of the metadata management efficiency, read / write time, storage space utilization, and fragmentation rate as the fitness of each file cluster corresponding to different capacity expansion granularities.

[0109] Furthermore, the first, second, third, and fourth weights corresponding to metadata management efficiency, read / write time, storage space utilization, and fragmentation rate can be obtained separately. The fitness of different capacity expansion granularities corresponding to each file cluster can also be obtained by weighted summation.

[0110] S360: The capacity expansion granularity with the highest adaptability is taken as the optimal capacity expansion granularity within the capacity expansion granularity range corresponding to each file cluster.

[0111] Specifically, the capacity expansion granularity with the highest adaptability within the capacity expansion granularity range is taken as the optimal capacity expansion granularity within the capacity expansion granularity range corresponding to each file cluster.

[0112] S370. Expand the capacity of the file directory where each file cluster is located based on the optimal capacity expansion granularity of each file cluster.

[0113] In this embodiment of the invention, numerical and non-numerical attribute information of each file in the virtual machine to be expanded is extracted from the file attribute information. The numerical attribute information is normalized to obtain the numerical feature vector of each file, and the non-numerical attribute information is one-hot encoded to obtain the binary feature vector of each file. A preset K-means clustering algorithm model is called, and the numerical feature vector and the binary feature vector are concatenated and input into the preset K-means clustering algorithm model to obtain at least one file cluster. The number of file clusters is the K value preset in the preset K-means clustering algorithm model. The space occupancy of each file cluster is obtained, and the capacity expansion granularity range of each file cluster is determined based on the space occupancy. A preset fitness evaluation index is obtained, and the fitness of each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters is determined based on the preset fitness evaluation index. The capacity expansion granularity with the highest fitness is taken as the optimal capacity expansion granularity within the capacity expansion granularity range corresponding to each file cluster. The capacity of the file directory where each file cluster is located is expanded based on the optimal capacity expansion granularity of each file cluster. This invention, by converting file attribute information into a vector of a unified format, eliminates limitations based on magnitude and type, ensuring the accuracy of file cluster partitioning. By quantitatively evaluating the merits of each capacity expansion granularity from multiple dimensions, it avoids performance imbalances caused by a single indicator, providing a comprehensive basis for selecting the optimal capacity expansion granularity and improving the accuracy of determining the optimal capacity expansion granularity.

[0114] Example 4

[0115] Figure 4 The following is a flowchart of another capacity expansion method provided in Embodiment 4 of the present invention. The embodiments of the present invention are optimizations of the above embodiments, specifically, supplementing another specific scenario of the capacity expansion process.

[0116] like Figure 4 As shown, another capacity expansion process includes: (1) obtaining the capacity utilization and fragmentation rate information of the large data virtual machine file system, classifying the small temporary files and log files in the virtual machine according to the analysis task type and dataset size, and obtaining the distribution of different file types.

[0117] The process involves acquiring the virtual machine file system's capacity utilization and fragmentation rate information. Files are initially categorized based on the analysis task type. Small temporary files and log files within the file system are classified, and file size, modification time, and access frequency are converted into numerical features. Non-numerical features such as file type are converted into numerical features using one-hot encoding. Files are then divided into different clusters to obtain the quantity distribution and space usage of each type of file. This includes using file system analysis tools to obtain basic information such as the total capacity, used capacity, and available capacity of the file system based on the virtual machine file system's capacity utilization and fragmentation rate information, and calculating the file system's capacity utilization. A file system fragmentation analysis tool scans the entire file system to obtain the physical distribution of files. By analyzing the continuity and dispersion of files, the file system's fragmentation rate is obtained. Based on file name, extension, path, and other information, files are initially classified using rule matching, dividing them into major categories such as system files, application files, and user data files, and obtaining the quantity and space usage of each type of file. By analyzing file attributes such as size, creation time, and modification time, heuristic rules are used to identify small temporary files and log files, classifying them separately to obtain the quantity and space usage of temporary and log files. Based on file size, modification time, and access frequency information, normalization is applied to convert these attributes into numerical features between 0 and 1, obtaining a numerical representation of the file. One-hot encoding is used to convert non-numerical attributes such as file type into binary feature vectors, obtaining a numerical representation of the file's non-numerical attributes. A pre-defined K-means clustering algorithm is used, combining the file's numerical and non-numerical features as input, and iterative calculations divide the files into different clusters, obtaining the clustering results. Based on the clustering results, the number of files and total size in each cluster are counted to obtain the distribution of file types and space usage.

[0118] Specifically, using file system analysis tools such as the `df` command, the total file system capacity is 500GB, the used capacity is 400GB, and the available capacity is 100GB, resulting in a file system capacity utilization rate of 80%. Scanning the entire file system using a file fragmentation analysis tool such as the `fsck` command reveals an average file continuity of 85% and a dispersion of 15%, resulting in a file system fragmentation rate of 15%. Based on file name, extension, path, and other information, files are initially classified using regular expression matching, categorized into system files, application files, and user data files. System files account for 20% of the total files, occupying 50GB of space; application files account for 30%, occupying 100GB of space; and user data files account for 50%, occupying 250GB of space. By analyzing file attributes such as size, creation time, and modification time, heuristic rules are used, such as identifying files smaller than 1MB with a creation time greater than 30 days from the current time as temporary files, and assigning them extensions. Files whose modification time differs from the current time by less than 7 days are identified as log files. The final number of temporary files is 10,000, occupying 1GB of space; the number of log files is 5,000, occupying 10GB of space. The file size, modification time, and access frequency attributes are normalized to their maximum and minimum values, resulting in numerical features between 0 and 1. The file type attribute is one-hot encoded, converting it into a binary feature vector with a length equal to the number of file types. The numerical and non-numerical features of the files are combined and input into a pre-defined K-means clustering algorithm model. By calculating the Euclidean distance between samples and iteratively updating the cluster centers, the files are ultimately divided into 5 clusters. The number of files and total size in each cluster are counted: Cluster 1 has 50,000 files and occupies 50GB of space; Cluster 2 has 80,000 files and occupies 200GB of space; Cluster 3 has 30,000 files and occupies 100GB of space; Cluster 4 has 20,000 files and occupies 30GB of space; and Cluster 5 has 20,000 files and occupies 20GB of space. This provides a comprehensive understanding of the virtual machine file system's capacity utilization, fragmentation, and the distribution characteristics of various file types, offering a basis for subsequent storage optimization and file management. Furthermore, when obtaining virtual machine file system information, the file system status was checked using the Linux system commands `df -h` and `fsck`, revealing a capacity utilization rate of 85% and a fragmentation rate of 12%. Based on the analysis task type, files were initially categorized into real-time processing and batch processing. For small temporary files and log files, a pre-defined K-means clustering algorithm was used for classification. First, file sizes were converted to KB units, modification times to hours from the current time, and access frequencies to the number of accesses within 24 hours.One-hot encoding was used for file types, such as [1,0,0] for text files, [0,1,0] for image files, and [0,0,1] for other files. The cluster size k was set to 3. After 10 iterations, the clustering results were obtained: the first cluster consisted of frequently accessed small files (30%), the second cluster consisted of frequently accessed large files (50%), and the third cluster consisted of frequently accessed medium-sized files (20%). Based on the clustering results, the e4defrag tool was used to defragment the files. First, the frequently accessed large files in the second cluster were processed, with a minimum file size threshold of 10MB and a fragmentation rate threshold of 5%. A scan found 100 files that met the criteria, with a total size of 500GB. The e4defrag tool reassembled these files, storing related file blocks contiguously. The fragmentation rate was reduced to 3%, and disk read / write speed was improved by 15%. A hierarchical storage mechanism was established for different file types. For the frequently accessed small files in the first cluster, the tmpfs memory file system was used for caching, allocating 1GB of memory space. For large files in the second cluster that are accessed infrequently, HDFS is used for storage, configured with 3 data nodes, each with a storage capacity of 2TB. Medium-sized files in the third cluster that are accessed in a medium-frequency manner are retained on the existing storage devices. Files not accessed for 30 days are automatically archived to low-speed storage devices, such as tape libraries. A scheduled task is set to perform file migration operations at 2 AM daily. When building the storage resource prediction model, a 7-day moving average method is used to predict short-term storage demand based on clustering results and fragmentation status. Storage usage data from the past 30 days is collected, and the average daily growth rate is calculated to be 0.5%. The predicted storage demand for the next 7 days is 103.5% of the current capacity. If the predicted result exceeds 90% of the storage capacity threshold, an automatic expansion mechanism is triggered, increasing the storage space by 10%. For example, if the current storage capacity is 10TB, and the predicted demand after 7 days is 10.35TB, exceeding the 9TB threshold, the system automatically adds 1TB of storage space to ensure the availability of the virtual machine file system.

[0119] (2) Based on the virtual machine operating system type and file system type, and combined with the distribution of different file types, quantitatively evaluate the impact of different expansion granularity schemes on the utilization of virtual machine storage resources, and determine the optimal expansion granularity threshold by combining the virtual disk format and storage resource pool capacity.

[0120] The system obtains information about the virtual machine's operating system type and file system type. The operating system type is obtained using the system command "uname -s," and the file system type is obtained using the command "df -T." Then, file system tools are used to scan the distribution of different file types. Files are classified according to size and access frequency. The K-means clustering algorithm is used to divide files into three categories: large (low-frequency access), medium (medium-frequency access), and small (high-frequency access), obtaining the number and total size of each category. For different file types, multiple expansion granularity schemes are set, increasing from 100MB to 1GB. Each scheme is simulated for expansion, and the storage resource utilization after expansion is calculated. The simulation considers file growth trends and access patterns. A multinomial regression algorithm is used to fit the relationship curve between expansion granularity and storage utilization to determine the optimal expansion granularity range. This includes classifying files into different categories such as structured data, unstructured data, and semi-structured data based on file type characteristics, obtaining the size distribution and access frequency distribution of each file type, providing a basis for subsequent expansion granularity settings. By analyzing the growth trends of different file types, the growth rate and patterns of each type of file were obtained. Combined with the current storage resource usage, the expansion granularity range for each file type was initially determined to be between 100MB and 1GB. A simulation method was used to gradually expand the storage resources for each file type in granularities from 100MB to 1GB, obtaining the usage of storage resources after expansion, including storage space utilization and file read / write performance, forming raw data on expansion granularity and resource utilization. Based on the raw data obtained during the simulated expansion process, a multinomial regression algorithm was used for curve fitting to obtain the functional relationship between expansion granularity and storage utilization, which was then visualized to intuitively reflect the non-linear correlation between the two. This included preprocessing the expansion granularity and storage utilization data using Python's NumPy library, and obtaining a standardized dataset through data cleaning and format conversion, preparing for subsequent multinomial regression fitting. Using the PolynomialFeatures class from Python's Scikit-learn library, the highest degree of the polynomial is set. The fit_transform method is used to transform the standardized data, obtaining the high-dimensional feature matrix required for polynomial regression. Based on the transformed high-dimensional feature matrix, the LinearRegression class from the Scikit-learn library is used, and the fit method is applied to fit the polynomial features and storage utilization, obtaining the coefficients and intercepts of the polynomial regression model, and establishing a nonlinear functional relationship between the expanded granularity and storage utilization.Using Python's NumPy library, a set of predicted storage utilization values ​​is generated using the `polyval` function based on the coefficients and intercept of the multinomial regression model. This yields the theoretical storage utilization corresponding to the expanded granularity within a given range, used to evaluate the model's fit. Using Python's Matplotlib library, the original data points and the multinomial regression fitting curve are visualized on the same coordinate system. By setting appropriate axis labels, legends, and titles, the non-linear relationship between expanded granularity and storage utilization is intuitively presented. By analyzing the morphological characteristics of the fitting curves, an expanded granularity range with high and stable storage resource utilization is determined as the optimal range for selecting expanded granularity, avoiding frequent expansion overhead from too small granularity and resource waste from too large granularity. Simultaneously, storage efficiency indicators and performance impact indicators for each scheme are calculated. Obtain virtual disk format information and storage resource pool capacity data. Combined with the optimal expansion granularity range obtained in the previous step, calculate the impact of different expansion granularities on the storage resource pool. Consider the characteristics of the virtual disk format, such as thin provisioning allowing dynamic allocation of storage space, while thick provisioning pre-allocates all space. Evaluate the impact of expansion operations on storage performance and select an expansion granularity that improves utilization without affecting performance. Based on the selected expansion granularity, set a dynamic storage resource utilization threshold. The threshold is automatically adjusted based on the current capacity of the storage resource pool and historical usage trends. By installing a storage monitoring agent in the virtual machine, storage usage data is collected every 5 minutes. When the utilization exceeds the dynamic upper limit threshold, automatic expansion is triggered; when the utilization is below the dynamic lower limit threshold for a certain period of time, automatic shrinkage is triggered, achieving dynamic adjustment and efficient utilization of storage resources.

[0121] Specifically, the system command "uname -s" was executed in the virtual machine to obtain the operating system type as Linux, and the file system type was obtained as ext4 using the command "df -T". File system scanning tools revealed that large, low-frequency access files accounted for 20% of the total, with a total size of 500GB; medium-sized, medium-frequency access files accounted for 40%, with a total size of 300GB; and small, high-frequency access files accounted for 40%, with a total size of 200GB. Using the K-means clustering algorithm with k=3 and 50 iterations, the files were divided into three categories, resulting in cluster centers of [10GB, 0.1 times / day], [1GB, 1 time / day], and [100MB, 10 times / day]. Ten expansion granularity schemes were set, ranging from 100MB to 1GB in 100MB increments. Simulated expansion was performed for each scheme, assuming a 1% daily increase in file size while maintaining a constant access frequency. A multinomial regression algorithm with degree=3 was used to fit the relationship between expansion granularity and storage utilization. Storage efficiency metrics (percentage increase in utilization) and performance impact metrics (percentage increase in I / O latency) were calculated. The storage efficiency metric for the 500MB expansion granularity scheme was 15%, and the performance impact metric was 5%. The virtual disk format was VMDK, and a thin provisioning mode was used. The total capacity of the storage resource pool was 10TB, and the current usage was 8TB. ​​Based on the results of multinomial regression, 700MB was selected as the optimal expansion granularity, achieving a balance between storage efficiency and performance impact. For thinly provisioned VMDK, expansion operations only allocate the space actually used, minimizing the impact on storage performance. A storage monitoring agent was installed in the virtual machine, configured to collect storage usage data every 5 minutes. A dynamic storage resource utilization threshold was set, with an initial upper limit of 85% and a lower limit of 60%. The threshold was automatically adjusted hourly based on the usage trend over the past 24 hours. When the utilization rate exceeds the upper limit threshold three times consecutively, automatic expansion was triggered, with each expansion being 700MB. When the utilization rate is below the lower limit threshold for six consecutive hours, automatic shrinkage was triggered, with each shrinkage being 350MB.

[0122] (3) Within the range of the expansion granularity threshold, fine-tune the expansion granularity, search for better solutions around the threshold, evaluate the impact of candidate solutions on small temporary files and log files, including metadata management efficiency, small I / O performance, storage space utilization and defragmentation effect, and find the optimal expansion granularity combination scheme that can balance the demand for small files and the utilization of storage resources.

[0123] Based on historical data analysis, the initial expansion granularity threshold range was determined to be 100MB to 1GB. Simulated annealing was used to search around this threshold, with an initial temperature of 100°C and a cooling coefficient of 0.95. Ten candidate solutions were generated per iteration, with a fine-tuning step size of 10MB. A fitness function value was calculated for each candidate solution using a weighted summation method, comprehensively considering four indicators: metadata management efficiency, small I / O performance, storage space utilization, and defragmentation effect, with weights of 0.3, 0.3, 0.2, and 0.2, respectively. For each candidate solution, the distribution of small temporary files and log files was scanned using the Linux fsstat tool to calculate the metadata management efficiency indicator. The response time and throughput of small I / O operations were measured using the fio tool to obtain the small I / O performance indicator. Storage space utilization was obtained using the df command, and the fragmentation rate was calculated using the e4defrag tool to obtain the defragmentation effect indicator. A balancing factor between small file access efficiency and overall storage utilization was also introduced. The four indicators are subjected to min-max normalization to calculate a comprehensive score. The candidate solution with the highest score is selected as the optimal solution for the current iteration. If a new solution is better than the current optimal solution, it is accepted with a certain probability, which decreases as the temperature decreases. This process is repeated until the maximum number of iterations is reached or the temperature drops to a preset threshold. The optimal expansion granularity combination scheme obtained by the search is verified by simulating read and write operations on a large number of small temporary files and log files in a test environment. Changes in metadata management efficiency, small I / O performance, storage space utilization, and defragmentation effect are monitored and compared with the evaluation results of the previous steps. If the preset performance indicators are met (e.g., a 10% improvement in metadata management efficiency, a 15% improvement in small I / O performance, a 5% increase in storage space utilization, and a 20% reduction in fragmentation rate), the scheme is adopted. Otherwise, the weights or search parameters are adjusted, and historical data is re-analyzed to obtain the optimal expansion granularity combination scheme that balances small file requirements and storage resource utilization.

[0124] (4) Based on the optimal expansion granularity combination scheme, expand the directory-level file system for directories where small temporary files and log files are concentrated in the virtual machine, predict the growth trend of small files in the future, expand specific directories in advance, and judge the storage resource utilization of the virtual machine.

[0125] Based on the optimal expansion granularity combination scheme, the Linux `du` command is used to perform a deep traversal of the directory structure in the virtual machine, counting the number, size, and access frequency of small temporary files and log files in each directory. The statistical results are stored in a database. K-means clustering algorithm is used to identify directories with concentrated file distribution, resulting in a directory-level file distribution heatmap. The heatmap uses color depth to represent file density, with red indicating high density and blue indicating low density. For the identified hot directories, the ARIMA model is used to process historical data, extracting the time features of file quantity, size, and access frequency. Combined with periodic and trend analysis, a predictive model is established to predict the growth trend of small files over the next 7 days, obtaining a predicted growth curve for each hot directory. Based on the predicted growth curve and the optimal expansion granularity scheme, the additional storage space required for each hot directory over the next 7 days is calculated. If the predicted growth exceeds 80% of the current remaining space, a directory-level file system expansion operation is triggered. The specific directory is expanded using file system management tools, with the expansion size being 120% of the predicted growth. If the expansion fails, it is automatically retried 3 times; if it still fails, an alarm message is sent. After implementing directory-level expansion, the storage usage of virtual machines is collected in real time through the Prometheus monitoring tool to calculate the overall storage resource utilization. If the utilization exceeds 90%, a global storage expansion process is triggered. If the utilization is below 30%, a storage resource reclamation process is triggered. Files that have not been accessed in the last 30 days are identified and moved to low-speed storage devices. Through dynamic adjustment, efficient utilization of virtual machine storage resources is ensured.

[0126] Specifically, in the virtual machine, a deep traversal of the directory structure was performed using the Linux `du` command. This revealed a large number of small log files in the ` / var / log` directory, occupying 2GB of space and exceeding 1 million files. Using the K-means clustering algorithm with k=5, the directory was clustered, identifying three hotspot directories: ` / var / log`, ` / tmp`, and ` / var / spool`. A heatmap showed ` / var / log` in dark red, indicating the highest file density. Analyzing the past 30 days of data using the ARIMA(2,1,2) model predicted a 100MB increase per day, or 20,000 files, for the ` / var / log` directory over the next 7 days. Based on the optimal expansion granularity of 280MB, the ` / var / log` directory was calculated to require an additional 840MB of storage space within 7 days. Since the predicted growth of 700MB exceeded 80% of the current remaining 500MB, a directory-level expansion operation was triggered, expanding ` / var / log` by 1008MB (120% of 840MB). During the expansion process, the file system management tool attempted the expansion three times, with the second attempt succeeding. The Prometheus monitoring tool collected storage usage data every 5 minutes, finding that the overall storage utilization rate was 85% after the expansion. After 24 hours, the storage utilization rate dropped to 75%, without triggering a global storage adjustment. Seven days later, executing the `du` command again revealed that the ` / var / log` directory had actually grown by 650MB and contained 17,000 files, with the error from the predicted result within 10%, verifying the accuracy of the prediction model.

[0127] (5) If the storage resource utilization of the virtual machine exceeds the preset threshold, file system-level expansion will be carried out. By analyzing the growth pattern of virtual machine business data and the support of snapshot technology, the entire file system will be expanded, and a data migration algorithm will be used to ensure data consistency during the expansion process.

[0128] The system collects virtual machine storage resource utilization data in real time using storage monitoring tools. When the utilization exceeds a preset threshold of 90%, a file system-level expansion process is triggered. An ARIMA model is used to model the business data growth over the past 30 days to predict the data growth for the next 7 days. Simultaneously, the Prophet algorithm is used to identify the periodicity and trend characteristics of data growth. Combined with the current remaining storage space, the required expansion capacity is calculated. Based on the file system type used by the virtual machine and the availability of snapshot technology, an appropriate file system expansion method is selected. A decision matrix based on the file system type (ext4, xfs, etc.) and virtualization platform (VMware, KVM, etc.) is used for judgment. If the file system supports online expansion, a logical volume manager is used to dynamically expand the file system size. If online expansion is not supported, a new storage volume is created and mounted to the virtual machine. Existing data is then migrated to the new storage volume using file system tools. During the expansion process, an rsync-based real-time synchronization mechanism is used for incremental replication, synchronizing data changes between the source and target file systems in real time. A write locking mechanism ensures data consistency. For large files requiring long-term migration, fixed-size 64MB data blocks are used for transmission. File metadata is updated after each data block migration to ensure recoverability during the migration process. After expansion, data integrity is verified using the fsck tool. If inconsistencies are found, an automatic repair process is triggered, including checking the file system structure, repairing damaged inodes and data blocks, and recovering lost files. Simultaneously, virtual machine storage configuration information is updated, file system parameters are adjusted to adapt to the new storage capacity, and storage resource utilization is recalculated. If the utilization rate is below a threshold, the expansion process is completed; otherwise, the aforementioned steps are repeated.

[0129] (6) During the file system expansion process, monitor the changes in the fragmentation rate of the virtual machine file system in real time. If the fragmentation rate exceeds the preset threshold, trigger the defragmentation mechanism. Use the hot data identification method to re-plan and reorganize the virtual disk storage space online, control the trend of fragmentation rate changes after expansion, and improve the utilization rate of virtual machine storage resources.

[0130] The Linux iostat tool collects virtual machine file system fragmentation data in real time. When the fragmentation rate exceeds a preset threshold of 60%, a defragmentation mechanism is triggered. Hadoop MapReduce analyzes the file access logs from the past 7 days to calculate the access popularity score for each file, marking the top 20% of files by access popularity score as hot data. Based on the hot data marking results, a greedy algorithm is used to re-plan the virtual disk storage space, sorting files in descending order of popularity score and allocating contiguous storage blocks accordingly. Hot data files are concentrated in contiguous storage blocks, while cold data files are stored in peripheral areas. File system metadata analysis tools identify severely fragmented files and prioritize defragmentation of these files, generating a file migration queue. An online data migration program is started, using a read-write separation approach to defragment the files in the migration queue. The migration buffer size is set to 64MB. Files larger than the buffer are migrated in blocks. After each file migration is completed, the file system metadata is updated. A semaphore mechanism limits the number of files migrated simultaneously to 5 to avoid excessive impact on virtual machine performance. During the cleanup process, the fragmentation rate trend is continuously monitored. The current fragmentation rate is calculated every time 10% of the files are migrated. A dynamic threshold is set, and the cleanup strategy is adjusted according to the cleanup progress. If the fragmentation rate drops below the preset threshold or reaches the maximum cleanup time limit of 4 hours, the cleanup process is stopped. Otherwise, file migration continues until all files are migrated or the stopping conditions are met. Finally, the virtual machine storage resource utilization statistics are updated.

[0131] Specifically, in the virtual machine environment, the iostat tool collects file system fragmentation data every 5 minutes. When the fragmentation rate reaches 65%, exceeding the preset threshold of 60%, the defragmentation mechanism is automatically triggered. A Hadoop MapReduce task is started, analyzing file access logs from the past 7 days (168 hours), involving 1 million files, and calculating the access popularity score for each file. The score calculation formula is: access count * 0.6 + recent access time weight * 0.4. After sorting, the top 200,000 files are marked as hot data. A greedy algorithm sorts the files in descending order of popularity score, allocating contiguous storage blocks to hot data files, starting from disk sector 0, with each file occupying adjacent sectors. File system metadata analysis identifies 150,000 severely fragmented files, which are prioritized for addition to the migration queue. The online data migration program is started, using 5 concurrent threads, each thread responsible for migrating one file. For files larger than 64MB, such as a 1GB database file, it is divided into 16 64MB blocks and migrated sequentially. During the migration, iostat recalculated the fragmentation rate every 10% of the files migrated, finding that the fragmentation rate dropped from 65% to 58%. The dynamic threshold adjustment strategy raised the stop threshold from the initial 50% to 55%, accelerating the defragmentation process. After 3 hours and 20 minutes of defragmentation, the fragmentation rate dropped to 54%, below the dynamic threshold, and the defragmentation process automatically stopped. Final statistics showed that storage resource utilization increased from 72% to 85%. Virtual machine services operated normally, and I / O performance improved by approximately 20%.

[0132] Example 5

[0133] Figure 5 This is a schematic diagram of another capacity expansion device provided in Embodiment 5 of the present invention. Figure 5 As shown, the device includes: a file cluster partitioning module 410, an interval determination module 420, a granularity determination module 430, and a capacity expansion module 440.

[0134] The file cluster partitioning module 410 is used to obtain the file attribute information of all files in the virtual machine to be expanded, and to partition all files into at least one file cluster based on the file attribute information.

[0135] The interval determination module 420 is used to obtain the space occupancy of each file cluster and the number of files contained in each file cluster, and to determine the capacity expansion granularity interval of each file cluster based on the space occupancy and the number of files.

[0136] Granularity determination module 430 is used to obtain preset fitness evaluation index and determine the optimal capacity expansion granularity of each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index.

[0137] The capacity expansion module 440 is used to expand the capacity of the file directory where each file cluster is located based on the optimal capacity expansion granularity of each file cluster.

[0138] Optionally, the file clustering module 410 is specifically used to extract the numerical and non-numerical attribute information of each file in the virtual machine to be expanded from the file attribute information; wherein, the numerical attribute information includes at least one of the following: file size, file modification time, and file access frequency; the non-numerical attribute information includes at least one of the following: file type and file storage location; the numerical attribute information is normalized to obtain the numerical feature vector of each file, and the non-numerical attribute information is one-hot encoded to obtain the binary feature vector of each file; a preset K-means clustering algorithm model is called, and the numerical feature vector and the binary feature vector are concatenated and input into the preset K-means clustering algorithm model to obtain at least one file cluster; wherein, the number of file clusters is the K value preset in the preset K-means clustering algorithm model.

[0139] Optionally, the interval determination module 420 includes: an interval acquisition unit, used to acquire an initial capacity expansion granularity interval pre-configured for each file cluster; wherein the initial capacity expansion granularity interval contains at least one initial capacity expansion granularity.

[0140] The utilization rate determination unit is used to simulate the expansion of the capacity of the file directory where each file cluster is located in a preset test environment according to different initial capacity expansion granularities within the initial capacity expansion granularity range, and determine the storage resource utilization rate of each file cluster after the simulated expansion based on the space occupancy.

[0141] The curve fitting unit is used to perform curve fitting between storage resource utilization and initial capacity expansion granularity for each file cluster using a multinomial regression algorithm, so as to obtain the relationship curve between storage resource utilization and initial capacity expansion granularity for each file cluster.

[0142] The interval extraction unit is used to extract high utilization intervals where the storage resource utilization rate is greater than a preset storage resource utilization rate threshold, as well as stable intervals of the relationship curve.

[0143] The interval determination unit is used to take the initial capacity expansion granularity range corresponding to the intersection of the high utilization interval and the stable interval as the capacity expansion granularity interval.

[0144] Optionally, the utilization determination unit is specifically used to obtain the initial total directory capacity of the file directory where each file cluster is located; to accumulate each initial capacity expansion granularity with the initial total directory capacity to obtain the total directory expansion capacity corresponding to each initial capacity expansion granularity after simulated expansion; to obtain the simulation test period under the preset test environment, and the total new space occupancy of each file cluster during the simulation test period; to obtain the sum of the total new space occupancy and the space occupancy, and to use the ratio of the sum to the total directory expansion capacity as the storage resource utilization rate corresponding to each initial capacity expansion granularity.

[0145] Optionally, the granularity determination module 430 includes: a fitness determination unit, used to obtain a preset fitness evaluation index and determine the fitness of each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index; and a granularity determination unit, used to take the capacity expansion granularity with the highest fitness as the optimal capacity expansion granularity within the capacity expansion granularity range corresponding to each file cluster.

[0146] Optionally, the fitness determination unit is specifically used to simulate the expansion of the capacity of the file directory where each file cluster is located in a preset test environment according to different capacity expansion granularities; according to preset fitness evaluation indicators, to obtain the metadata management efficiency, read and write time, storage space utilization and fragmentation rate of each file cluster after simulated expansion; and to use the normalized cumulative value of metadata management efficiency, read and write time, storage space utilization and fragmentation rate as the fitness of each file cluster corresponding to different capacity expansion granularities.

[0147] The capacity expansion device provided in the embodiments of the present invention can execute the capacity expansion method provided in any embodiment of the present invention, and has the corresponding beneficial effects of executing the method.

[0148] Example 6

[0149] Embodiment 6 of the present invention provides an electronic device for performing a capacity expansion method, a computer-readable storage medium, and a computer.

[0150] Figure 6 A schematic diagram of an electronic device is shown that can be used to implement any of the capacity expansion methods of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown in the embodiments of the present invention, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the embodiments of the present invention described and / or claimed herein.

[0151] like Figure 6 As shown, the electronic device includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for device operation. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0152] Multiple components in the electronic device are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as capacity expansion methods.

[0154] In some embodiments, the capacity expansion method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the capacity expansion method may be performed. Alternatively, in other embodiments, processor 11 may be configured as the capacity expansion method by any other suitable means (e.g., by means of firmware).

[0155] Various embodiments of the systems and technologies described above in these embodiments of the present invention can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input system, and at least one output system, and transmitting data and instructions to the storage system, the at least one input system, and the at least one output system.

[0156] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing system, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0157] In the context of embodiments of the present invention, a computer-readable storage medium may be a tangible storage medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, system, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal storage medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display system (e.g., a cathode ray tube or liquid crystal display monitor) for displaying information to the user; and a keyboard and pointing system (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of systems can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including: sound input, voice input, or tactile input).

[0159] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or storage medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0160] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system. This addresses the shortcomings of traditional physical hosts and virtual private servers, such as high management difficulty and weak business scalability.

[0161] It should be understood that the various forms of procedures shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and no limitation is imposed herein.

[0162] The specific embodiments described above do not constitute a limitation on the scope of protection of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A capacity expansion method, characterized in that, The method includes: Obtain the file attribute information of all files in the virtual machine to be expanded, and divide all files into at least one file cluster based on the file attribute information; Obtain the space occupancy of each file cluster, and determine the capacity expansion granularity range of each file cluster based on the space occupancy; Obtain a preset fitness evaluation index, and determine the optimal capacity expansion granularity for each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index. The capacity of the file directory containing each file cluster is expanded based on the optimal capacity expansion granularity for each file cluster.

2. The method according to claim 1, characterized in that, The step of obtaining file attribute information of all files in the virtual machine to be expanded, and dividing all files into at least one file cluster based on the file attribute information, includes: Extract the numerical and non-numerical attribute information of each file in the virtual machine to be expanded from the file attribute information; wherein, the numerical attribute information includes at least one of the following: file size, file modification time, and file access frequency; the non-numerical attribute information includes at least one of the following: file type and file storage location; The numerical attribute information is normalized to obtain the numerical feature vector of each file, and the non-numerical attribute information is one-hot encoded to obtain the binary feature vector of each file. A preset K-means clustering algorithm model is invoked. The numerical feature vector and the binary feature vector are concatenated and then input into the preset K-means clustering algorithm model to obtain at least one file cluster. The number of file clusters is the K value preset in the preset K-means clustering algorithm model.

3. The method according to claim 1, characterized in that, The step of determining the capacity expansion granularity range for each file cluster based on the space occupancy includes: Obtain the initial capacity expansion granularity range pre-configured for each of the file clusters; wherein, the initial capacity expansion granularity range contains at least one initial capacity expansion granularity; In a preset test environment, the capacity of the file directory where each file cluster is located is simulated to be expanded one by one according to different initial capacity expansion granularities within the initial capacity expansion granularity range, and the storage resource utilization rate of each file cluster after the simulated expansion is determined based on the space occupancy. For each of the file clusters, a multinomial regression algorithm is used to fit the storage resource utilization rate with the initial capacity expansion granularity to obtain the relationship curve between the storage resource utilization rate and the initial capacity expansion granularity for each of the file clusters. From the relationship curve, extract the high utilization range where the storage resource utilization rate is greater than the preset storage resource utilization rate threshold, and the stable range of the relationship curve; The initial capacity expansion granularity range corresponding to the intersection of the high utilization range and the stable range is taken as the capacity expansion granularity range.

4. The method according to claim 3, characterized in that, In a preset test environment, the capacity of the file directory containing each file cluster is simulated and expanded one by one according to different initial capacity expansion granularities within the initial capacity expansion granularity range. The storage resource utilization rate of each file cluster after the simulated expansion is determined based on the space occupancy. This includes: Obtain the initial total directory capacity of the file directories where each of the aforementioned file clusters is located; The initial capacity expansion granularity is added to the initial total directory capacity to obtain the total directory expansion capacity corresponding to each initial capacity expansion granularity after simulated expansion; Obtain the simulation test period under the preset test environment, and the total new space occupancy of each file cluster within the simulation test period; Obtain the sum of the total newly added space and the total space occupied, and use the ratio of the sum to the total capacity of the directory expansion as the storage resource utilization rate corresponding to each initial capacity expansion granularity.

5. The method according to claim 1, characterized in that, The step of obtaining a preset fitness evaluation index and determining the optimal capacity expansion granularity for each file cluster within the capacity expansion granularity range corresponding to different file clusters includes: Obtain a preset fitness evaluation index, and determine the fitness of each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index. The capacity expansion granularity with the highest fitness is taken as the optimal capacity expansion granularity within the capacity expansion granularity range corresponding to each file cluster.

6. The method according to claim 5, characterized in that, The preset fitness evaluation metrics include at least: metadata management efficiency metrics, read / write performance metrics, storage space utilization metrics, and fragmentation rate metrics. Determining the fitness of each capacity expansion granularity within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation metrics includes: In a preset test environment, the capacity of the file directory containing each file cluster is simulated and expanded one by one according to different capacity expansion granularities; According to the preset fitness evaluation index, obtain the metadata management efficiency, read / write time, storage space utilization and fragmentation rate of each file cluster after simulated expansion; The sum of the normalized values ​​of the metadata management efficiency, read / write time, storage space utilization, and fragmentation rate is used as the fitness factor for different capacity expansion granularities corresponding to each file cluster.

7. A capacity expansion device, characterized in that, The device includes: The file cluster partitioning module is used to obtain the file attribute information of all files in the virtual machine to be expanded, and to partition all files into at least one file cluster based on the file attribute information; The interval determination module is used to obtain the space occupancy of each file cluster and the number of files contained in each file cluster, and to determine the capacity expansion granularity interval of each file cluster based on the space occupancy and the number of files. The granularity determination module is used to obtain a preset fitness evaluation index and determine the optimal capacity expansion granularity of each file cluster within the capacity expansion granularity range corresponding to different file clusters based on the preset fitness evaluation index. The capacity expansion module is used to expand the capacity of the file directory where each file cluster is located based on the optimal capacity expansion granularity of each file cluster.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the capacity expansion method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the capacity expansion method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes: A computer program that, when executed by a processor, implements the capacity expansion method according to any one of claims 1-6.

Images