Distributed object storage demand evaluation method and device, electronic equipment, storage medium and program product

Abstract

The application provides a distributed object storage demand evaluation method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of distributed object storage. After obtaining the user demand, the demand is decomposed into performance demand, capacity space demand and data protection demand. The required HDD hard disk quantity is calculated according to the performance demand, the object quantity is calculated according to the storage capacity demand, and the required index pool space and DB total space are calculated, and the required SSD disk quantity is obtained after conversion. The best server quantity is calculated in combination with the hard disk collocation of different servers and the data protection demand. Thus, through in-depth analysis of the performance demand, capacity demand and data protection demand of the user and conversion of the demand into a specific parameterized model, the actual demand of the user can be accurately analyzed, the demand matching degree is improved, and more importantly, the cost benefit is considered, so that the designed configuration scheme meets the demand and is also economical and efficient.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of distributed object storage, and particularly relate to a distributed object storage demand evaluation method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] Distributed object storage is a storage solution that achieves high availability, high performance and scalability by storing data on multiple storage nodes. The service provided by object storage is a nearly unlimited capacity large file KV storage. It is suitable for storing unstructured data such as pictures, text, audio and video. For video monitoring and large video storage applications, user needs are usually diverse. How to design the optimal server configuration scheme according to different types of user needs is still a challenge.

[0003] When selecting a distributed object storage server, users usually propose corresponding storage requirements, which generally include performance requirements and capacity requirements. Existing demand configuration schemes often rely on experience or simple linear expansion, which not only cannot accurately meet the specific needs of users, but also may lead to resource waste or performance bottlenecks. Moreover, it also increases server costs and later operation costs.

[0004] Therefore, there is an urgent need for a new distributed object storage demand evaluation method. SUMMARY

[0005] Embodiments of the present application provide a distributed object storage demand evaluation method, device, electronic equipment, storage medium and program product to at least partially solve the above problems.

[0006] The first aspect of the embodiments of the present application provides a distributed object storage demand evaluation method, which comprises:

[0007] Obtaining storage demand information, the storage demand information comprising: storage capacity demand, performance demand and data protection demand; the storage capacity demand representing the storage space of a distributed object storage cluster, the performance demand representing the read-write bandwidth information and read-write IOPS information of the distributed object storage cluster, and the data protection demand representing the redundancy ratio of the data pool of the distributed object storage cluster;

[0008] According to the storage capacity demand, determining the number of objects to be stored;

[0009] Based on the number of objects, the pre-determined index pool space occupation value, the DB space occupation value, and the capacity of a single block SSD disk, determining the estimated number of SSD disks;

[0010] determining an estimated HDD disk quantity according to the performance requirement and the read bandwidth average, the write bandwidth average, the read IOPS average and the write IOPS average of each HDD disk;

[0011] determining a minimum server quantity based on the estimated SSD disk quantity and the estimated HDD disk quantity;

[0012] adjusting the minimum server quantity based on the data protection requirement to obtain an estimated server quantity;

[0013] determining a single-disk capacity of the HDD disk according to the storage capacity requirement, the estimated server quantity and the HDD disk quantity of a single server;

[0014] The index pool space occupation value, the DB space occupation value and the read bandwidth average, the write bandwidth average, the read IOPS average and the write IOPS average of each HDD disk are obtained through testing on a distributed object storage test cluster.

[0015] Optionally, the object quantity to be stored is determined according to the storage capacity requirement, including:

[0016] In a case where the storage capacity requirement includes an object average size requirement, the object quantity to be stored is determined according to the storage capacity requirement and the object average size requirement;

[0017] In a case where the object average size requirement is not included in the storage capacity requirement, the object average size is estimated according to a storage scene corresponding to the storage requirement, and the object quantity to be stored is determined according to the storage capacity requirement and the estimated object average size.

[0018] Optionally, the estimated server quantity is obtained by adjusting the minimum server quantity based on the data protection requirement, including:

[0019] In a case where it is determined in the data protection requirement that EC erasure code is used for data protection;

[0020] determining a data block quantity N and a check block quantity M;

[0021] In a case where the minimum server quantity is less than N+M, the estimated server quantity is determined as N+M+1;

[0022] In a case where the minimum server quantity is greater than or equal to N+M, the estimated server quantity is determined as the minimum server quantity.

[0023] Optionally, the estimated server quantity is obtained by adjusting the minimum server quantity based on the data protection requirement, including:

[0024] In a case where it is determined in the data protection requirement to employ a replica mode for data protection;

[0025] The estimated server quantity is determined as a product of the minimum server quantity and the replica quantity.

[0026] Optionally, the single-disk capacity of the HDD disk is determined according to the storage capacity requirement, the estimated server quantity, and the HDD hard disk quantity of a single server, comprising:

[0027] The data pool raw capacity is determined according to the storage capacity requirement, the data block quantity N, the check block quantity M, and a preset capacity usage alarm threshold.

[0028] The single-disk capacity of the HDD disk is determined according to the data pool raw capacity, the estimated server quantity, and the HDD hard disk quantity of a single server.

[0029] Optionally, the index pool space occupation value, the DB space occupation value, and the preset read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk are determined by the following steps:

[0030] In the distributed object storage test cluster, an index pool is created using an SSD disk, and a data pool with a preset redundancy ratio is created using an HDD disk;

[0031] In the test environment, a preset number of objects with a preset size are uploaded to a storage bucket through a client connection to an RGW gateway, the index pool usage capacity before uploading, the DB space already used by each OSD, and the capacity occupied by the index pool after uploading, the DB space occupied by each OSD are collected, and the index pool space occupation value and the DB space occupation value of each object with the preset size are determined;

[0032] In the distributed object storage test cluster, the read-write bandwidth and read-write IOPS of the HDD disk are obtained to obtain the preset read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk;

[0033] The index pool space occupation value, the DB space occupation value of the object with the preset size, and the read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk are saved;

[0034] The method further comprises:

[0035] According to the redundancy ratio and the average object size, the index pool space occupation value, the DB space occupation value of the object with the corresponding size tested in the distributed object storage test cluster with the corresponding redundancy ratio, and the read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk are obtained.

[0036] The second aspect of the embodiment of the present application provides a distributed object storage demand evaluation device, a demand decomposition module is used to obtain storage demand information, the storage demand information includes: storage capacity demand, performance demand and data protection demand; the storage capacity demand represents storage space of a distributed object storage cluster, the performance demand represents read-write bandwidth information and read-write IOPS information of the distributed object storage cluster, and the data protection demand represents a redundancy ratio of a data pool of the distributed object storage cluster;

[0037] A demand conversion module is used to determine the number of objects that need to be stored according to the storage capacity demand, determine an estimated number of SSD disks based on the number of objects, a pre-determined index pool space occupation value, a DB space occupation value and the capacity of a single SSD disk, determine an estimated number of HDD disks according to the performance demand and the average read bandwidth, the average write bandwidth, the average read IOPS and the average write IOPS of each HDD disk, determine a minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks, adjust the minimum number of servers based on the data protection demand to obtain an estimated number of servers, and determine the single-disk capacity of the HDD disk according to the storage capacity demand, the estimated number of servers and the number of HDD hard disks of a single server.

[0038] The index pool space occupation value, the DB space occupation value and the average read bandwidth, the average write bandwidth, the average read IOPS and the average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

[0039] The third aspect of the embodiment of the present application provides an electronic device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the distributed object storage demand evaluation method of the first aspect of the present application when executed.

[0040] The fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the distributed object storage demand evaluation method of the first aspect of the present application.

[0041] The fifth aspect of the embodiment of the present application provides a computer program product, which includes a computer program / instruction, and the computer program / instruction is used to implement the steps in the distributed object storage demand evaluation method of the first aspect of the present application by a processor.

[0042] In the embodiment of the present application, the required basic data is obtained in advance by calculation on the test cluster, including the space occupied by the index data of each object in the index pool, the DB space occupied by the metadata of each object, and the limit read-write bandwidth and limit read-write IOPS that can be reached by the single-block HDD hard disk in the cluster. After obtaining the user requirements, the requirements are decomposed into performance requirements, capacity space requirements and data protection requirements. According to the performance requirements, the required number of HDD hard disks can be calculated. If the average size requirement of the objects stored in the cluster is given, the number of objects that need to be stored in the cluster can be calculated according to the space requirement. If the average size of the objects is not given, a recommended range is given according to the user cluster usage scenario. Then, according to the calculated number of objects, the total required index pool space and DB total space are calculated, and after conversion, the required number of SSD disks is obtained. Combined with the hard disk collocation of different servers and the data protection requirements, the best number of storage servers is calculated.

[0043] The distributed object storage requirement evaluation method provided by the embodiment of the present application can accurately analyze the actual requirements of the user by deeply analyzing different requirements of the user, such as performance requirements, capacity requirements and data protection requirements, and converting these requirements into specific parameterized models, and can improve the requirement matching degree. More importantly, the cost-effectiveness is taken into account to ensure that the designed configuration scheme not only meets the requirements, but also is economical and efficient.

[0044] In addition, the technical scheme provided by the embodiment of the present application can also improve the service customization capability of the distributed object storage system, facilitate the operation team to provide more personalized and optimized configuration schemes for different users, fully exert the performance of the storage server, promote the efficient use of server resources, reduce resource waste, and also has important significance for reducing operation cost and improving system efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0046] Figure 1 is a step flow chart of the distributed object storage requirement evaluation method provided by the embodiment of the present application;

[0047] Figure 2 is a detailed flowchart of the distributed object storage requirement evaluation method provided by the embodiment of the present application;

[0048] Figure 3is a basic data determination step flow chart in a distributed object storage demand evaluation method provided by an embodiment of the application;

[0049] Figure 4 is a distributed object storage demand evaluation device structure schematic diagram provided by an embodiment of the application. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0051] Generally, when a user selects a distributed object storage server, the user will propose corresponding storage demands, which generally include performance demands and capacity demands. If the two types of demand information are simply relied on, the experience value or linear expansion through stacked servers is used to meet the demands, which will largely cause the actual performance to not meet the demands or the cost to increase, and cause difficulties in the later cluster operation and maintenance.

[0052] In an embodiment of the application, the demands of the user are deeply analyzed, a parameterized demand model is constructed, and the performance demands, capacity demands and data protection demands of the user are converted into quantifiable parameters. Then, through a series of mathematical calculations, the user demands are accurately matched.

[0053] Specifically, an embodiment of the application provides a distributed object storage demand evaluation method, as shown in Figure 1 The method comprises the following steps:

[0054] S101, storage demand information is acquired, and the storage demand information comprises storage capacity demands, performance demands and data protection demands.

[0055] The storage capacity demands represent the storage space of a distributed object storage cluster, the performance demands represent read-write bandwidth information and read-write IOPS information of the distributed object storage cluster, and the data protection demands represent the redundancy ratio of a data pool of the distributed object storage cluster.

[0056] S102, the number of objects that need to be stored is determined according to the storage capacity demands.

[0057] S103, the number of estimated SSD disks is determined based on the number of objects, a pre-determined index pool space occupation value, a DB space occupation value and the capacity of a single block SSD disk.

[0058] S104, the number of estimated HDD disks is determined according to the performance demands and the read bandwidth average value, the write bandwidth average value, the read IOPS average value and the write IOPS average value of each HDD disk.

[0059] S105, determining a minimum server number based on the estimated SSD disk number and the estimated HDD disk number.

[0060] S106, adjusting the minimum server number based on the data protection requirement to obtain an estimated server number.

[0061] S107, determining a single-disk capacity of an HDD disk according to the storage capacity requirement, the estimated server number and the HDD disk number of a single server.

[0062] The index pool space occupation value, the DB space occupation value and the read bandwidth average value, the write bandwidth average value, the read IOPS average value and the write IOPS average value of each HDD disk are obtained by testing on a distributed object storage test cluster.

[0063] Firstly, the terms involved in the embodiments of the present application are explained as follows:

[0064] The index pool (Index pool) in the distributed object storage is mainly used for saving the object list in the storage bucket, and is also used for storing the index data of the storage bucket, and is generally created using an SSD disk.

[0065] The data pool (Data pool) in the distributed object storage is mainly used for storing the object data uploaded by the user. Generally, an HDD disk is used to create it.

[0066] RGW: full name RADOS GateWay, is the interface provided by the distributed object storage for object storage services.

[0067] rados object: when the user uploads an object to the RGW storage bucket, the object data will be divided into multiple data blocks and distributed on different OSDs (Object Storage Daemon) in the distributed object storage cluster. One RGW object contains one or more rados objects, and the size of each rados object is 4MB.

[0068] DB: mainly refers to the RocksDB space allocated to each OSD. RocksDB is a storage engine library based on a key-value storage interface, in which the key and value are arbitrary byte streams. The extended attributes of each uploaded object are stored in the RocksDB database in the form of Key-value.

[0069] In the embodiment of the present application, the basic user requirements generally include performance requirements, capacity requirements, detailed user needs, data protection methods required by the user, and the average size of objects stored in the cluster. In the case where the user requirements are not clear, the basic object size can be estimated in combination with the user's distributed object storage cluster usage scenario. The performance requirements indicate the read-write bandwidth and read-write IOPS information of the storage cluster. The capacity requirements indicate the at least required storage space of the cluster, including the storage space of large objects and small objects. The data protection mode indicates that the data access service will not be affected in the case of failure, and the replica or EC erasure code mode can be used.

[0070] In the embodiment of the present application, after the user's requirements are decomposed in detail, the requirements can be converted into various parameter data. Specifically, in combination with Figure 2 The distributed object storage requirement evaluation method provided by the embodiment of the present application is explained and described.

[0071] In the embodiment of the present application, the number of disks is first calculated. Specifically, the step S102 includes the following sub-steps:

[0072] S1021, in the case where the storage capacity requirement includes the object average size requirement, determining the number of objects to be stored according to the storage capacity requirement and the object average size requirement.

[0073] S1022, in the case where the storage capacity requirement does not include the object average size requirement, estimating the object average size according to the storage scene corresponding to the storage requirement, and determining the number of objects to be stored according to the storage capacity requirement and the estimated object average size.

[0074] In the embodiment of the present application, whether the storage capacity requirement is included in the user input storage requirement can be determined. In the case of inclusion, whether the object average size requirement is included is further determined. In the case of non-inclusion, the estimated object average size is determined according to the business model.

[0075] In the embodiment of the present application, the estimation of the object average size can use any mature technical means in related technologies. For example, the estimation can be performed according to the business model and the user required storage scene (for example, storage of monitoring video).

[0076] In the embodiment of the present application, the number of objects to be stored can be determined according to the storage capacity requirement and the object average size. Further, the total capacity of the index pool space and the total capacity of the DB space required can be determined according to the number of objects and the pre-determined index pool space occupation value and DB space occupation value.

[0077] In the embodiment of the present application, the pre-determined index pool space occupation value and DB space occupation value can be estimated values corresponding to the average size of the objects, that is, different sizes of objects can correspond to different index pool space occupation values and DB space occupation values, for example, objects of sizes of 4KB, 32KB, 256KB, 4MB, 10MB, 22MB and 34MB respectively occupy index pool space occupation values and DB space occupation values. Correspondingly, the corresponding index pool space occupation value and DB space occupation value can be determined according to the average size of the objects.

[0078] In the embodiment of the present application, the pre-determined index pool space occupation value and DB space occupation value can also be an estimated general value, that is, different sizes of objects correspond to the same index pool space occupation value and DB space occupation value. For example, objects of sizes of 4KB, 32KB, 256KB, 4MB, 10MB, 22MB and 34MB correspond to the same index pool space occupation value V index and DB space occupation value V db .

[0079] In the embodiment of the present application, in the user storage capacity demand, there can also be a subdivision of large object storage demand (including: large object storage capacity demand and large object average size), and small object storage demand (including: small object storage capacity demand and small object average size).

[0080] In the embodiment of the present application, considering that the size of a single rados object is the default 4MB, therefore in the embodiment of the present application, a large object refers to an object with a capacity greater than 4MB, which needs to be divided into multiple data blocks and stored on different OSDs in the cluster. A small object refers to an object with a capacity less than or equal to 4MB, which does not need to be divided into multiple data blocks and can be directly stored on different OSDs in the cluster.

[0081] In the embodiment of the present application, in this case, the total capacity of the required index pool space and the total capacity of the required DB space can be respectively calculated according to the large object storage demand and the small object storage demand respectively.

[0082] Specifically, the number of large objects = large object storage capacity demand ÷ large object average size, and the number of small objects = small object storage capacity demand ÷ small object average size.

[0083] The total capacity of the index pool space = (the number of large objects + the number of small objects) × V index × 3. In the embodiment, three copies of data safe index pool are used.

[0084] The total capacity of the DB space = [(the number of large objects × large object average size ÷ 4MB) + the number of small objects] × V dbAmong them, 4MB represents the capacity size of the rados object.

[0085] In the embodiment of the application, in each step of calculating the quantity, the integer is rounded up.

[0086] In an actual production environment, a piece of SSD disk is created into multiple DB partitions for multiple OSDs to use, based on which, in the embodiment of the application, the estimated SSD disk quantity is equal to (total capacity of the index pool space + total capacity of the DB space) divided by the number of DB partitions divided by the capacity of a single piece of SSD disk.

[0087] Specifically, since the bandwidth of the distributed object storage cluster mainly depends on the number and performance of the HDD hard disk, the number of HDD hard disks can be calculated according to the performance requirement.

[0088] In the embodiment of the application, considering that the read-write bandwidth of the distributed object storage cluster mainly depends on the read-write bandwidth and number of the HDD disk, and the read-write IOPS of the distributed object storage cluster is found to mainly depend on the read-write IOPS and number of the HDD disk after testing, the two types of basic data can obtain the limit value in the test environment. The average read bandwidth of each HDD hard disk is R bandwidth , the average write bandwidth is W bandwidth , the corresponding read IOPS is R iops , and the write IOPS is W iops .

[0089] In the embodiment of the application, the performance requirement of the user can include read-write bandwidth requirement value and read-write IOPS requirement value.

[0090] Correspondingly, the number of write HDD disks is equal to the write bandwidth requirement value divided by W bandwidth , and the number of read HDD disks is equal to the read bandwidth requirement value divided by R bandwidth In order to ensure the maximum performance, the maximum value of the two quantities can be obtained as the number of HDD disks.

[0091] The number of write HDD disks is equal to the write IOPS requirement value divided by W iops , and the number of read HDD disks is equal to the read IOPS requirement value divided by R iops In order to ensure the maximum performance, the maximum value of the two quantities can be obtained as the number of HDD disks.

[0092] Further, based on the larger one of the number of HDD disks determined according to the read-write bandwidth requirement value and the number of HDD disks determined according to the read-write IOPS requirement value, the estimated number of HDD disks is determined.

[0093] Further, in the embodiment of the application, the types of the server are divided into capacity type, performance type and standard type, the number of SSD and HDD disks matched on each type of server is different, resulting in different prices of the servers. Therefore, in addition to correctly and reasonably estimating the number of servers according to actual demand, the final cost demand also needs to be considered.

[0094] Therefore, in the embodiment of the application, the number of different types of servers required can be determined according to the number of SSD and HDD disks matched on different types of servers, and the calculated estimated number of SSD disks and the estimated number of HDD disks, and then the total price can be obtained according to the number of different types of servers and the corresponding price, and then the scheme with the lowest cost is determined, thereby determining the type of server and the corresponding minimum number of servers. For example, a standard 12-disk server is selected, matched with 2 SSD disks + 10 HDD, and the minimum number of servers is equal to max (SSS disk number ÷ 2, HDD disk number ÷ 10).

[0095] In the embodiment of the application, the calculation of the hard disk capacity of the data pool first confirms the data protection mode adopted in the data protection demand of the user, and then determines the redundancy ratio of the data pool of the distributed object storage cluster.

[0096] In the case where the user does not specify the data protection demand, or does not specify the specific N+M number or the number of copies of EC, the selection can be made according to the storage scene and the user storage demand.

[0097] Specifically, the step S106 includes the following two cases:

[0098] In the case where it is determined in the data protection demand that EC error correction code is used for data protection: the number of data blocks N and the number of check blocks M are determined; in the case where the minimum number of servers is less than N+M, the estimated number of servers is determined as N+M+1; in the case where the minimum number of servers is greater than or equal to N+M, the estimated number of servers is determined as the minimum number of servers.

[0099] In the case where it is determined in the data protection demand that the copy mode is used for data protection: the estimated number of servers is determined as the product of the minimum number of servers and the number of copies.

[0100] In the embodiment of the application, the EC error correction code adopts a 4K alignment strategy, and needs to meet the user's requirement of ensuring the overall safety of the system under the failure condition, in addition, the EC error correction code also needs to meet the condition that the number of storage servers is greater than N+M. According to the calculated minimum number of servers, the configuration with the highest cost performance in different EC ratios is selected.

[0101] For example, the minimum server quantity is calculated as 5, and the copy mode is determined in the data protection requirement of the user, so that the closest and cost-effective EC configuration ratio mode is 4+2, but in order to meet the requirement of N+M, the minimum server quantity is changed to 7; if the minimum server quantity is calculated as 8, the redundancy ratio of EC 4+2 is configured, and adjustment is not required.

[0102] In the case that the copy mode is determined in the data protection requirement of the user, the estimated server quantity is equal to the minimum server quantity multiplied by the copy quantity.

[0103] In the embodiment of the application, after the estimated server quantity is determined based on the data protection scheme, the disk capacity can be further determined.

[0104] Specifically, the step S107 includes the following sub-steps:

[0105] S1071, determining the data pool raw capacity according to the storage capacity requirement, the data block quantity N, the check block quantity M and the preset capacity usage alarm threshold.

[0106] S1072, determining the single-disk capacity of the HDD disk according to the data pool raw capacity, the estimated server quantity and the HDD hard disk quantity of a single server.

[0107] In the embodiment of the application, the data pool raw capacity is equal to the storage space capacity requirement multiplied by (N+M) divided by N divided by the preset capacity usage alarm threshold, and the preset capacity usage alarm threshold can be 0.85.

[0108] The data pool single-disk capacity selection is equal to the data pool raw capacity divided by the minimum server quantity divided by the HDD hard disk quantity of a single server. The single-disk capacity of the HDD disk is generally 12T or 16T, and the closest capacity can be selected according to the calculation result.

[0109] The capacity of a single SSD disk is generally 960G, 1.92T or 3.2T, and in the case that the cost can be controlled, a high-performance nvme SSD can also be selected to create an index pool.

[0110] After a series of calculations, the final estimated server quantity is determined to meet the overall capacity space requirement, the object quantity requirement that can be stored, and the data protection requirement. However, in order to achieve the best performance, the number of application servers that match the performance of the application servers and the network bandwidth allowed conditions are also selected. The performance of the storage server can be fully utilized, and the utilization rate of the server is improved.

[0111] In the embodiment of the application, the number of distributed object storage cluster buckets can be further determined.

[0112] In the distributed object storage cluster, the storage bucket can simultaneously accept a large number of read / write / delete operations, and it is necessary to control the upper limit of the number of objects in each storage bucket at the business level without affecting the performance, so as to avoid overflow and cause performance jitter. Therefore, the number of storage buckets to be created = (number of large objects + number of small objects) ÷ maximum number of objects per bucket. When it is monitored that the number of objects written to the storage bucket approaches the threshold, traffic control or traffic switching can be performed to write the current main business to other storage buckets or perform dispersed writing. The upper layer business is ensured to be unaware.

[0113] Based on the above embodiment, in the embodiment of the application, the estimated number of servers, the number of SSD and HDD disks matched by each server, the single disk capacity of the HDD disk, the single disk capacity of the SSD disk, and the number of storage buckets can be determined, so that the configuration scheme of the distributed object storage cluster meeting the user's demand can be determined by deeply analyzing the different needs of the user, such as performance demand, capacity demand and data protection demand.

[0114] In the embodiment of the application, the index pool space occupation value, the DB space occupation value, and the preset read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk are determined by the following steps:

[0115] S1, in the distributed object storage test cluster, an index pool is created using an SSD disk, and a data pool with a preset redundancy ratio is created using an HDD disk.

[0116] S2, in the test environment, a preset number of objects with a preset size are uploaded to the storage bucket through the client connection RGW gateway, the used capacity of the index pool before uploading, the used DB space of each OSD, and the occupied capacity of the index pool after uploading, the occupied DB space of each OSD are collected, and the index pool space occupation value and the DB space occupation value of each object with the preset size are determined.

[0117] S3, in the distributed object storage test cluster, the read / write bandwidth and read / write IOPS of the HDD disk are obtained, and the preset read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk are obtained.

[0118] S4, the index pool space occupation value and the DB space occupation value of the object with the preset size, and the read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk are saved.

[0119] In the embodiment of the present application, considering that in the distributed object storage, the object uploaded by the client into the storage cluster will generate three types of data, the index data will be saved in the index pool, the metadata information will be saved in the DB space allocated to each OSD, and the data will be saved to the data pool. Therefore, the three spaces occupied by each object in the storage cluster can be calculated / prepared in advance.

[0120] Specifically, in order to facilitate understanding, combined with the Figure 3 The basic data determination step in the distributed object storage demand evaluation method proposed in the embodiment of the present application is explained and described.

[0121] In the distributed object storage test cluster, an index pool with three-copy data security is created using an SSD disk. The redundancy ratio used in the data pool may be different due to different usage scenarios and data protection requirements. Therefore, on the test cluster, multiple data pools with different redundancy ratios can be created for testing. Thus, the index pool space occupation value, DB space occupation value, and the average read bandwidth, write bandwidth, read IOPS, and write IOPS of the HDD disk of the object under different redundancy ratios are obtained.

[0122] During the test process, the client uploads objects to the storage bucket by connecting the RGW gateway. In order to better and more accurately obtain the actual space size occupied by each object, the objects can be uploaded multiple times under the condition that the available capacity of the cluster allows, and at least 10 million objects of the same size are uploaded to the cluster each time. Taking the uploading of 10 million 1MB objects to a cluster with an EC 4+2 redundancy ratio as an example for description:

[0123] Before uploading, record the index pool usage capacity, data pool usage capacity, and DB space occupied by each OSD in the cluster. After uploading is completed, record the capacity occupied by the index pool, the capacity occupied by the data pool, and the capacity occupied by each OSD again. Then, calculate the difference value of the three spaces, and then calculate the three basic data results in the following manner:

[0124] The space occupied by each object in the index pool = index pool capacity increment ÷ 10 million

[0125] The space occupied by each object in the data pool = data pool capacity increment ÷ 10 million

[0126] The space occupied by each object in the DB = sum of all OSD DB increments ÷ rados object number ÷ 6

[0127] According to the above calculation method, the actual space size occupied by objects of sizes such as 4KB, 32KB, 256KB, 4MB, 10MB, 22MB, and 34MB can be calculated, respectively.

[0128] Through actual calculation, it is found that objects of different sizes occupy index pool space in the cluster basically consistently, and the estimated fixed value V can be used in subsequent user demand conversion index Estimation is performed.

[0129] In addition, the space occupied by objects of different sizes in the DB also remains within ±20% of the fixed value V db , and the fixed value V can be approximately used in subsequent user demand conversion db Estimation is performed.

[0130] In terms of performance, the read-write bandwidth of the cluster is mainly determined by the read-write bandwidth and number of HDD disks, and the read-write IOPS of the cluster is mainly determined by the read-write IOPS and number of HDD disks after testing. Both of these two types of basic data can obtain the limit value in the test environment. Assuming that the average read bandwidth of each HDD disk is R bandwidth, , the average write bandwidth is W bandwidth , the corresponding read IOPS is R iops , and the write IOPS is Wiops .

[0131] Therefore, several types of basic data required for demand conversion can be obtained in advance and archived, and do not need to be obtained again in subsequent demand evaluation.

[0132] S100, according to the redundancy ratio and the average size of the object, obtaining the index pool space occupation value, the DB space occupation value, and the read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk of the corresponding size of the object in the distributed object storage test cluster corresponding to the redundancy ratio.

[0133] In the embodiment of the application, step S100 can be executed after step S101 and before step S103.

[0134] In the embodiment of the application, in the actual application process, the pre-stored index pool space occupation value, DB space occupation value, and read bandwidth average value, write bandwidth average value, and read IOPS average value and write IOPS average value of the HDD disk of the corresponding size of the object in the distributed object storage test cluster corresponding to the redundancy ratio can be obtained according to the redundancy ratio and the average size of the object in the user demand. Therefore, in the subsequent demand conversion process, the most accurate estimation result is obtained.

[0135] In the embodiment of the application, in the actual application process, a unified fixed value can also be directly used for calculation to simplify the estimation process.

[0136] Based on the same inventive concept, the embodiment of the present application also provides a distributed object storage demand evaluation device, the device comprises:

[0137] a demand decomposition module, configured to acquire storage demand information, wherein the storage demand information comprises storage capacity demand, performance demand and data protection demand; the storage capacity demand represents storage space of a distributed object storage cluster; the performance demand represents read-write bandwidth information and read-write IOPS information of the distributed object storage cluster; and the data protection demand represents a redundancy ratio of a data pool of the distributed object storage cluster;

[0138] a demand conversion module, configured to determine an object quantity to be stored according to the storage capacity demand; determine an estimated SSD disk quantity based on the object quantity, a pre-determined index pool space occupation value, a DB space occupation value and a capacity of a single block SSD disk; determine an estimated HDD disk quantity according to the performance demand and a read bandwidth average value, a write bandwidth average value, a read IOPS average value and a write IOPS average value of each HDD disk; determine a minimum server quantity based on the estimated SSD disk quantity and the estimated HDD disk quantity; adjust the minimum server quantity based on the data protection demand to obtain an estimated server quantity; and determine a single disk capacity of an HDD disk according to the storage capacity demand, the estimated server quantity and a HDD hard disk quantity of a single server.

[0139] The index pool space occupation value, the DB space occupation value and the read bandwidth average value, the write bandwidth average value, the read IOPS average value and the write IOPS average value of each HDD disk are obtained through testing on a distributed object storage test cluster.

[0140] In an optional implementation of the embodiment of the present application, the demand conversion module comprises an object quantity determination unit, configured to:

[0141] In a case where the storage capacity demand comprises an object average size demand, the object quantity to be stored is determined according to the storage capacity demand and the object average size demand.

[0142] In a case where the storage capacity demand does not comprise the object average size demand, an object average size is estimated according to a storage scene corresponding to the storage demand, and the object quantity to be stored is determined according to the storage capacity demand and the estimated object average size.

[0143] In an optional implementation of the embodiment of the present application, the demand conversion module comprises a server quantity adjustment unit, configured to:

[0144] In a case where the data protection demand determines that an EC error correction code is used for data protection;

[0145] determining a data block quantity N and a check block quantity M;

[0146] in a case where the minimum server quantity is less than N+M, determining an estimated server quantity as N+M+1;

[0147] in a case where the minimum server quantity is greater than or equal to N+M, determining the estimated server quantity as the minimum server quantity.

[0148] In an optional implementation of an embodiment of the present application, the server quantity adjustment unit is further configured to:

[0149] in a case where it is determined in the data protection requirement that a copy mode is used for data protection;

[0150] determining the estimated server quantity as a product of the minimum server quantity and a copy quantity.

[0151] In an optional implementation of an embodiment of the present application, the requirement conversion module comprises a capacity determination unit configured to:

[0152] determining a data pool raw capacity according to the storage capacity requirement, the data block quantity N, the check block quantity M, and a preset capacity usage alarm threshold;

[0153] determining a single-disk capacity of an HDD disk according to the data pool raw capacity, the estimated server quantity, and a single-server HDD disk quantity.

[0154] In an optional implementation of an embodiment of the present application, the index pool space occupation value, the DB space occupation value, and the preset read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk are determined by the following steps:

[0155] In a distributed object storage test cluster, an index pool is created using an SSD disk, and a data pool with a preset redundancy ratio is created using an HDD disk;

[0156] In a test environment, a client is connected to an RGW gateway to upload a preset quantity of objects with a preset size to a storage bucket, and the index pool usage capacity before uploading, the DB space already used by each OSD, and the capacity occupied by the index pool after uploading, the DB space occupied by each OSD are collected to determine the index pool space occupation value and the DB space occupation value of each object with the preset size;

[0157] In the distributed object storage test cluster, the read-write bandwidth and the read-write IOPS of the HDD disk are obtained to obtain the preset read bandwidth average value, the write bandwidth average value, and the read IOPS average value and the write IOPS average value of the HDD disk;

[0158] save the index pool space occupation value, the DB space occupation value and the HDD disk average read bandwidth, write bandwidth, read IOPS and write IOPS of the object of the preset size;

[0159] The device further comprises a basic data acquisition unit configured to acquire, according to the redundancy ratio and the object average size, the index pool space occupation value, the DB space occupation value and the HDD disk average read bandwidth, write bandwidth, read IOPS and write IOPS of the object of the corresponding size obtained by testing in the distributed object storage test cluster corresponding to the redundancy ratio.

[0160] Specifically, Figure 4 A structure diagram of a distributed object storage demand evaluation device provided by an embodiment of the present application is shown in FIG. 1. Figure 4 As shown in FIG. 1, in the embodiment of the present application, the basic data acquisition module can acquire, from the pre-calculated and saved basic data, the index pool space occupation of the object, the DB space occupation of the underlying rados object, the standard read-write bandwidth and IOPS (HDD disk average read bandwidth, write bandwidth, read IOPS and write IOPS) of the HDD disk under different IO granularities and the upper limit of the number of objects that can be stored in the storage bucket.

[0161] The index pool space occupation of the object, the DB space occupation of the underlying rados object, the standard read-write bandwidth and IOPS of the HDD disk under different IO granularities are used to determine the optimal configuration scheme of the storage server, and the upper limit of the number of objects that can be stored in the storage bucket is used to determine the minimum number of storage buckets in the storage cluster.

[0162] In the embodiment of the present application, the demand decomposition module can analyze the storage capacity requirement in the storage requirement input by the user, determine whether the requirement specification of the capacity and the object size is contained, and then calculate the number of objects that need to be stored by the demand conversion module based on the requirement specification, and calculate the number of SSD disks in combination with the basic data acquired by the basic data acquisition module.

[0163] The demand decomposition module can also analyze the performance requirement in the storage requirement input by the user, determine whether the read-write bandwidth and IOPS requirement is contained, and then calculate the number of HDD disks by the demand conversion module based on the read-write bandwidth and IOPS requirement in combination with the basic data acquired by the basic data acquisition module. The demand decomposition module can also analyze the data protection requirement in the storage requirement input by the user, and then the demand conversion module can determine the optimal configuration scheme of the storage server according to the calculated number of SSD disks and the number of HDD disks in combination with the data protection requirement.

[0164] Based on the above embodiment, the customer's demand can be deeply parameterized and decomposed based on the demand decomposition module to design a configuration scheme that better meets the customer's demand, thereby improving the demand matching degree.

[0165] Based on the above embodiment, comprehensive performance analysis and cost-benefit analysis can be integrated, and through accurate demand analysis and multi-dimensional optimization design, the efficient use of server resources can be promoted, and resource waste can be reduced. It is of great significance to reduce operating costs and improve system efficiency.

[0166] Based on the above embodiment, the customization capability of the systematic distributed object storage cluster can be realized, which greatly enhances the customization service capability of the distributed object storage system. In addition, the future scalability and maintainability of the system are considered, and it is ensured that the configuration scheme can easily adapt to future technology upgrades or demand changes.

[0167] In summary, the technical scheme provided by the embodiment of the application can significantly improve the utilization rate and cost performance of storage resources by accurately matching user demand and optimizing the configuration scheme, save server operation costs, and also customize storage services according to different needs of different customers. It is particularly useful for enterprises in different industries and of different sizes, and can meet diversified and specific storage needs.

[0168] Based on the same inventive concept, the embodiment of the application also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes, the steps in the distributed object storage demand evaluation method according to any of the above embodiments are implemented.

[0169] Based on the same inventive concept, the embodiment of the application also provides a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement the steps in the distributed object storage demand evaluation method according to any of the above embodiments.

[0170] Based on the same inventive concept, the embodiment of the application provides a computer program product, which includes computer programs / instructions that are executed by a processor to implement the steps in the distributed object storage demand evaluation method according to any of the above embodiments.

[0171] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0172] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, apparatus, or computer program product. Accordingly, embodiments of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0173] Embodiments of the present application are described herein with reference to the drawings, which are as follows: Figure 1 Figure 1 The computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart block or blocks.

[0174] The computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 The computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart block or blocks.

[0175] The computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 The computer program instructions can also be loaded onto a computer or other programmable terminal device to cause a series of operational steps to be performed on the computer or other programmable terminal device to produce a computer-implemented process such that the instructions which execute on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowchart block or blocks.

[0176] While preferred embodiments of the present application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such variations and modifications as fall within the scope of the present application.

[0177] ​​​Finally, it is to be understood that the phraseology or terminology such as "first" and "second" etc. used herein is merely intended to differentiate one entity or operation from another entity or operation, without necessarily requiring or implying any actual such relationship or order between such entities or operations. Moreover, the terms "comprising", "including", or any other closure, are intended to cover the non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include those elements alone but can include other elements not expressly listed or even include elements inherent in such process, method, article, or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element.

[0178] The above describes in detail the distributed object storage demand assessment method provided by the present application, and the principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A method for evaluating the requirements of distributed object storage, characterized in that, The method comprises: acquiring storage requirement information, the storage requirement information comprising: storage capacity requirement, performance requirement and data protection requirement; the storage capacity requirement representing storage space of a distributed object storage cluster, the performance requirement representing read-write bandwidth information and read-write IOPS information of the distributed object storage cluster, and the data protection requirement representing redundancy ratio of a data pool of the distributed object storage cluster; determining the number of objects to be stored according to the storage capacity requirement; determining the estimated number of SSD disks based on the number of objects, a predetermined index pool space occupancy value, a DB space occupancy value and the capacity of a single SSD disk; determining the estimated number of HDD disks according to the performance requirement and the average read bandwidth, the average write bandwidth, the average read IOPS and the average write IOPS of each HDD disk; determining the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks; adjusting the minimum number of servers based on the data protection requirement to obtain the estimated number of servers; determining the single-disk capacity of HDD disks according to the storage capacity requirement, the estimated number of servers and the number of HDD hard disks of a single server. The index pool space occupancy value, the DB space occupancy value and the average read bandwidth, the average write bandwidth, the average read IOPS and the average write IOPS of each HDD disk are obtained through testing on a distributed object storage test cluster.

2. The method of claim 1, wherein, The method comprises: in the case that the storage capacity requirement comprises object average size requirement, determining the number of objects to be stored according to the storage capacity requirement and the object average size requirement; in the case that the storage capacity requirement does not comprise object average size requirement, estimating the object average size according to the storage scenario corresponding to the storage requirement, and determining the number of objects to be stored according to the storage capacity requirement and the estimated object average size.

3. The method of claim 1, wherein, The method comprises: in the case that the data protection requirement determines to use EC erasure code for data protection; determining the number of data blocks N and the number of check blocks M; in the case that the minimum number of servers is less than N+M, determining the estimated number of servers as N+M+1; in the case that the minimum number of servers is greater than or equal to N+M, determining the estimated number of servers as the minimum number of servers.

4. The method of claim 1, wherein, The method comprises: in the case that the data protection requirement determines to use a replication mode for data protection; determining the estimated number of servers as the product of the minimum number of servers and the number of replicas.

5. The method of claim 3, wherein, The method comprises: determining the data pool raw capacity according to the storage capacity requirement, the number of data blocks N, the number of check blocks M and a preset capacity usage alarm threshold. Determine the single-disk capacity of the HDD disk according to the data pool bare capacity, the estimated server number, and the HDD hard disk number of a single server.

6. The method of claim 1-5, wherein, The index pool space occupation value, the DB space occupation value, and the preset read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk are determined by the following steps: In the distributed object storage test cluster, use the SSD disk to create the index pool and use the HDD disk to create the data pool with the preset redundancy ratio; In the test environment, connect the RGW gateway through the client to upload a preset number of objects with a preset size to the storage bucket, collect the index pool usage capacity before uploading, the DB space already used by each OSD, and the index pool occupation capacity after uploading, the DB space occupied by each OSD, and determine the index pool space occupation value, the DB space occupation value of each object with the preset size, and the read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk; In the distributed object storage test cluster, obtain the read-write bandwidth and read-write IOPS of the HDD disk to obtain the read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk; Save the index pool space occupation value, the DB space occupation value of the object with the preset size, and the read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk; The method further comprises: According to the redundancy ratio and the average object size, obtain the index pool space occupation value, the DB space occupation value of the object with the corresponding size obtained in the distributed object storage test cluster with the corresponding redundancy ratio, and the read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of the HDD disk.

7. A distributed object storage demand assessment apparatus, characterized by, The apparatus comprises: A demand decomposition module is configured to obtain storage demand information, wherein the storage demand information comprises storage capacity demand, performance demand, and data protection demand; the storage capacity demand represents the storage space of a distributed object storage cluster, the performance demand represents the read-write bandwidth information and read-write IOPS information of the distributed object storage cluster, and the data protection demand represents the redundancy ratio of the data pool of the distributed object storage cluster; A demand conversion module is configured to determine the number of objects to be stored according to the storage capacity demand, determine the estimated number of SSD disks based on the number of objects, the index pool space occupation value, the DB space occupation value, and the capacity of a single SSD disk, determine the estimated number of HDD disks according to the performance demand and the read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of each HDD disk, determine the minimum server number based on the estimated number of SSD disks and the estimated number of HDD disks, adjust the minimum server number based on the data protection demand to obtain the estimated server number, and determine the single-disk capacity of the HDD disk according to the storage capacity demand, the estimated server number, and the HDD hard disk number of a single server. The index pool space occupation value, the DB space occupation value, and the read bandwidth average value, the write bandwidth average value, the read IOPS average value, and the write IOPS average value of each HDD disk are obtained by testing on a distributed object storage test cluster.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the distributed object storage demand evaluation method of any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the distributed object storage demand evaluation method of any one of claims 1-6.

10. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the steps in the distributed object storage demand evaluation method of any one of claims 1-6.

Images