Distributed object storage demand evaluation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a distributed object storage demand evaluation method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of distributed object storage. After a user demand is obtained, the demand is decomposed into a performance demand, a capacity space demand and a data protection demand. The number of required HDDs is calculated according to performance requirements, the number of objects is calculated according to storage capacity requirements, then required index pool space and DB total space are calculated, and the number of required SSDs is obtained after conversion. And calculating the optimal number of servers in combination with hard disk matching and data protection requirements of different servers. Therefore, by deeply analyzing the performance requirement, the capacity requirement and the data protection requirement of the user and converting the requirements into the specific parameterized model, the actual requirement of the user can be accurately analyzed, the requirement matching degree is improved, more importantly, the cost benefit is considered, and it is ensured that the designed configuration scheme meets the requirements and is economical and efficient.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of distributed object storage technology, and in particular to a distributed object storage demand assessment method, device, electronic device, storage medium and program product. Background Art

[0002] Distributed Object Storage is a storage solution that achieves high availability, high performance, and scalability by distributing data across multiple storage nodes. The service provided by object storage is a large file KV storage with nearly unlimited capacity. It is suitable for storing unstructured data such as pictures, texts, audio and video. For actual application scenarios such as video surveillance and large video storage, user needs are usually diverse. How to design the optimal server configuration solution based on different types of user needs is still a challenge.

[0003] When users select a distributed object storage server, they usually put forward corresponding storage requirements, which generally include performance requirements and capacity requirements. Existing demand configuration solutions often rely on experience judgment or simple linear expansion methods, which not only cannot accurately meet the specific needs of users, but may also lead to resource waste or performance bottlenecks. It will also increase server costs and subsequent operating costs.

[0004] Therefore, a new distributed object storage demand assessment method is urgently needed. Summary of the invention

[0005] Embodiments of the present invention provide a distributed object storage demand assessment method, device, electronic device, storage medium and program product to at least partially solve the above-mentioned problems.

[0006] A first aspect of an embodiment of the present invention provides a method for evaluating distributed object storage requirements, the method comprising:

[0007] Acquire storage demand information, the storage demand information including: storage capacity demand, performance demand and data protection demand; the storage capacity demand represents the storage space of the distributed object storage cluster, the performance demand represents the read and write bandwidth information and read and write IOPS information of the distributed object storage cluster, and the data protection demand represents the redundancy ratio of the data pool of the distributed object storage cluster;

[0008] Determine the number of objects that need to be stored according to the storage capacity requirement;

[0009] Determine the estimated number of SSD disks based on the number of objects, the predetermined index pool space occupancy value, the DB space occupancy value, and the capacity of a single SSD disk;

[0010] Determine the estimated number of HDD disks based on performance requirements and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk;

[0011] Determine the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks;

[0012] Based on the data protection requirements, the minimum number of servers is adjusted to obtain an estimated number of servers;

[0013] Determine the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers and the number of HDD hard disks of a single server;

[0014] The index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

[0015] Optionally, determining the number of objects to be stored according to the storage capacity requirement includes:

[0016] In a case where the storage capacity requirement includes an average object size requirement, determining the number of objects to be stored according to the storage capacity requirement and the average object size requirement;

[0017] When the storage capacity requirement does not include the average object size requirement, the average object size is estimated according to the storage scenario corresponding to the storage requirement, and the number of objects to be stored is determined according to the storage capacity requirement and the estimated average object size.

[0018] Optionally, based on the data protection requirement, the minimum number of servers is adjusted to obtain an estimated number of servers, including:

[0019] In the case where it is determined in the data protection requirement that EC erasure code is used for data protection;

[0020] Determine the number of data blocks N and the number of check blocks M;

[0021] When the minimum number of servers is less than N+M, the estimated number of servers is determined to be N+M+1;

[0022] When the minimum number of servers is greater than or equal to N+M, the estimated number of servers is determined to be the minimum number of servers.

[0023] Optionally, based on the data protection requirement, the minimum number of servers is adjusted to obtain an estimated number of servers, including:

[0024] In the case where it is determined in the data protection requirement that a replica mode is used for data protection;

[0025] The estimated number of servers is determined as the product of the minimum number of servers and the number of replicas.

[0026] Optionally, determining the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers, and the number of HDD hard disks of a single server includes:

[0027] Determine the raw capacity of the data pool according to the storage capacity requirement, the number of data blocks N, the number of check blocks M and the preset capacity usage alarm threshold;

[0028] The single disk capacity of the HDD disk is determined according to the raw capacity of the data pool, the estimated number of servers, and the number of HDD hard disks of a single server.

[0029] Optionally, the index pool space occupancy value, the DB space occupancy value, and the preset read bandwidth average value, write bandwidth average value, read IOPS average value, and write IOPS average value of the HDD disk are determined by adopting the following steps:

[0030] In the distributed object storage test cluster, use SSD disks to create an index pool, and use HDD disks to create a data pool with a preset redundancy ratio;

[0031] In the test environment, upload a preset number of objects of preset size to the storage bucket through the client connection to the RGW gateway, collect the index pool usage capacity before uploading, the DB space used by each OSD, and the capacity occupied by the index pool and the DB space occupied by each OSD after uploading, and determine the index pool space occupancy value and DB space occupancy value of each object of the preset size;

[0032] In the distributed object storage test cluster, the read and write bandwidth and read and write IOPS of the HDD disk are obtained, and the preset read bandwidth average, write bandwidth average, read IOPS average and write IOPS average of the HDD disk are obtained;

[0033] Save the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the object of the preset size;

[0034] The method further comprises:

[0035] According to the redundancy ratio and the average object size, the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of objects of corresponding sizes obtained by testing in the distributed object storage test cluster with the corresponding redundancy ratio are obtained.

[0036] A second aspect of an embodiment of the present invention provides a distributed object storage demand assessment device, a demand decomposition module, which is used to obtain storage demand information, wherein the storage demand information includes: storage capacity demand, performance demand and data protection demand; the storage capacity demand represents the storage space of the distributed object storage cluster, the performance demand represents the read and write bandwidth information and read and write IOPS information of the distributed object storage cluster, and the data protection demand represents the redundancy ratio of the data pool of the distributed object storage cluster;

[0037] The demand conversion module is used to determine the number of objects that need to be stored according to the storage capacity requirement; determine the estimated number of SSD disks based on the number of objects, the predetermined index pool space occupancy value, the DB space occupancy value, and the capacity of a single SSD disk; determine the estimated number of HDD disks according to the performance requirements and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk; determine the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks; adjust the minimum number of servers based on the data protection requirements to obtain the estimated number of servers; determine the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers, and the number of HDD hard disks of a single server;

[0038] The index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

[0039] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when executed, the processor implements the distributed object storage demand assessment method as described in the first aspect of the present invention.

[0040] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed object storage demand assessment method as described in the first aspect of the present invention.

[0041] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is used by a processor to implement the steps in the distributed object storage demand assessment method described in the first aspect of the present invention.

[0042] In an embodiment of the present invention, the required basic data is obtained in advance through calculation on the test cluster, including the space occupied by the index data of each object in the index pool, the DB space occupied by the metadata of each object, and the maximum read and write bandwidth and maximum read and write IOPS that can be achieved by a single HDD hard disk in the cluster. After obtaining the user's needs, the needs are decomposed into performance requirements, capacity space requirements, and data protection requirements. According to the performance requirements, the number of HDD hard disks required can be calculated. If the average size requirement of the objects stored in the cluster is given, the number of objects that the cluster needs to store can be calculated based on the space requirements. If the average size of the objects is not given, a recommended range is given according to the user's cluster usage scenario. Based on the calculated number of objects, the total required index pool space and DB total space are calculated, and the number of SSD disks required can be obtained after conversion. Combined with the hard disk matching of different servers and data protection requirements, the optimal number of storage servers is calculated.

[0043] The distributed object storage demand assessment method provided by the embodiment of the present invention can not only accurately analyze the actual needs of users, but also improve the matching degree of needs by deeply analyzing different needs of users, such as performance needs, capacity needs and data protection needs, and converting these needs into specific parameterized models. More importantly, it takes into account cost-effectiveness, ensuring that the designed configuration solution is economical and efficient while meeting the needs.

[0044] In addition, the technical solution provided by the embodiment of the present invention can also improve the service customization capability of the distributed object storage system, making it easier for the operation team to provide more personalized and optimized configuration solutions for different users. It can give full play to the performance of the storage server, promote the efficient use of server resources, and reduce resource waste. It is also of great significance to reduce operating costs and improve system efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0046] Figure 1 is a flowchart of the steps of a distributed object storage demand assessment method provided by an embodiment of the present invention;

[0047] Figure 2 It is a detailed flow chart of a distributed object storage demand assessment method provided by an embodiment of the present invention;

[0048] Figure 3It is a flow chart of the basic data determination step in the distributed object storage demand assessment method provided by an embodiment of the present invention;

[0049] Figure 4 It is a schematic diagram of the structure of a distributed object storage demand assessment device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0051] Usually, when users select a distributed object storage server, they will propose corresponding storage requirements, which generally include performance requirements and capacity requirements. If you simply rely on these two types of demand information, use experience or linearly expand by stacking servers to meet the requirements, it will largely cause the actual performance to not meet the requirements or increase costs, causing difficulties in the later operation and maintenance of the cluster.

[0052] In the embodiment of the present invention, the user's needs are deeply analyzed, a parameterized demand model is constructed, and the user's performance requirements, capacity requirements and data protection requirements are converted into quantifiable parameters. Then, after a series of mathematical calculations, the user's needs are accurately matched.

[0053] Specifically, an embodiment of the present invention provides a distributed object storage demand assessment method, such as Figure 1 As shown, the method comprises the following steps:

[0054] S101, obtaining storage requirement information, where the storage requirement information includes: storage capacity requirement, performance requirement and data protection requirement.

[0055] The storage capacity requirement represents the storage space of the distributed object storage cluster, the performance requirement represents the read and write bandwidth information and read and write IOPS information of the distributed object storage cluster, and the data protection requirement represents the redundancy ratio of the data pool of the distributed object storage cluster.

[0056] S102: Determine the number of objects that need to be stored according to the storage capacity requirement.

[0057] S103, determining the estimated number of SSD disks based on the number of objects, a predetermined index pool space occupancy value, a DB space occupancy value, and the capacity of a single SSD disk.

[0058] S104: Determine an estimated number of HDD disks according to the performance requirement and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk.

[0059] S105: Determine the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks.

[0060] S106: Based on the data protection requirement, adjust the minimum number of servers to obtain an estimated number of servers.

[0061] S107, determining the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers, and the number of HDD hard disks of a single server.

[0062] The index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

[0063] First, the terms involved in the embodiments of the present invention are explained as follows:

[0064] The index pool in distributed object storage is mainly used to save the object list in the bucket and also to store the index data of the bucket. It is generally created using SSD disks.

[0065] The data pool in the distributed object storage is mainly used to store the object data uploaded by users. It is usually created using HDD disks.

[0066] RGW: The full name is RADOS GateWay, which is the interface for distributed object storage to provide object storage services to the outside world.

[0067] rados object: When a user uploads an object to an RGW bucket, the object data will be split into multiple data blocks and distributed on different OSDs (Object Storage Daemons) in the distributed object storage cluster. An RGW object contains one or more rados objects, and the size of each rados object is 4MB.

[0068] DB: Mainly refers to the RocksDB space allocated to each OSD. RocksDB is a storage engine library based on a key-value pair storage interface, where the key and value are arbitrary byte streams. The extended attributes of each uploaded object will be stored in the RocksDB database in the form of key-value.

[0069] In the embodiment of the present invention, the basic user requirements generally include performance requirements, capacity requirements, detailed user requirements, the data protection method proposed by the user, and the average size of objects stored in the cluster. When the user requirements are not clear, the basic object size can be estimated in combination with the user's distributed object storage cluster usage scenario. Among them, the performance requirements indicate the read and write bandwidth of the storage cluster, as well as the read and write IOPS information. The capacity requirements indicate the minimum storage space that the cluster needs to provide, including storage space for large objects and small objects. The data protection mode will indicate that the data access service will not be affected in the event of a failure, and a replica or EC erasure code mode can be used.

[0070] In the embodiment of the present invention, after the user's needs are decomposed in detail, the needs can be converted into various parameter data. Specifically, Figure 2 The distributed object storage demand assessment method provided in an embodiment of the present invention is explained.

[0071] In the embodiment of the present invention, the number of disks is first calculated. Specifically, step S102 includes the following sub-steps:

[0072] S1021: When the storage capacity requirement includes an average object size requirement, determine the number of objects that need to be stored according to the storage capacity requirement and the average object size requirement.

[0073] S1022, when the storage capacity requirement does not include the average object size requirement, estimate the average object size according to the storage scenario corresponding to the storage requirement, and determine the number of objects that need to be stored according to the storage capacity requirement and the estimated average object size.

[0074] In an embodiment of the present invention, the storage requirements input by the user can be used to determine whether they include storage capacity requirements. If so, it can be further determined whether they include object average size requirements. If not, an estimate is made based on the business model to determine the estimated average object size.

[0075] In the embodiment of the present invention, the average size of the object can be estimated by any mature technical means in the relevant technology. For example, the average size of the object can be estimated according to the business model and the storage scenario required by the user (for example, the storage of surveillance videos).

[0076] In the embodiment of the present invention, the number of objects to be stored can be determined according to the storage capacity requirement and the average size of the objects, and the total capacity of the index pool space and the total capacity of the DB space required can be determined according to the number of objects and the predetermined index pool space occupancy value and DB space occupancy value.

[0077] In the embodiment of the present invention, the predetermined index pool space occupancy value and DB space occupancy value may be an estimated value corresponding to the average size of the object, that is, objects of different sizes may correspond to different index pool space occupancy values ​​and DB space occupancy values, for example, the index pool space occupancy values ​​and DB space occupancy values ​​occupied by objects of 4KB, 32KB, 256KB, 4MB, 10MB, 22MB, and 34MB sizes, respectively. Accordingly, the corresponding index pool space occupancy value and DB space occupancy value may be determined according to the average size of the object.

[0078] In the embodiment of the present invention, the predetermined index pool space occupancy value and DB space occupancy value may also be an estimated universal value, that is, objects of different sizes correspond to the same index pool space occupancy value and DB space occupancy value. For example: objects of 4KB, 32KB, 256KB, 4MB, 10MB, 22MB, and 34MB sizes correspond to the same index pool space occupancy value V index 、DB space occupancy value V db .

[0079] In an embodiment of the present invention, the user storage capacity requirement may be further subdivided into large object storage requirements (including: large object storage capacity requirements and average size of large objects), and small object storage requirements (including: small object storage capacity requirements and average size of small objects).

[0080] In the embodiment of the present invention, considering that the size of a single rados object is 4MB by default, in the embodiment of the present invention, a large object refers to an object with a capacity greater than 4MB, which needs to be divided into multiple data blocks and stored on different OSDs in the cluster. A small object refers to an object with a capacity less than or equal to 4MB, which does not need to be divided into multiple data blocks and can be directly stored on different OSDs in the cluster.

[0081] In the embodiment of the present invention, in this case, the required total index pool space capacity and the required total DB space capacity may be calculated according to the large object storage demand and the small object storage demand respectively.

[0082] Specifically, the number of large objects = large object storage capacity requirements ÷ average size of large objects, and the number of small objects = small object storage capacity requirements ÷ average size of small objects.

[0083] Total index pool space capacity = (number of large objects + number of small objects) × V index ×3. In this embodiment, a three-copy data-secure index pool is used.

[0084] Total DB space capacity [(number of large objects × average size of large objects ÷ 4MB) + number of small objects] × V db. Among them, 4MB represents the capacity of the rados object.

[0085] In the embodiment of the present invention, in each step of calculating the quantity, an integer is rounded upwards.

[0086] In an actual production environment, one SSD disk will be created into multiple DB partitions for use by multiple OSDs. Based on this, in an embodiment of the present invention, the estimated number of SSD disks = (total capacity of index pool space + total capacity of DB space ÷ number of DB partitions) ÷ the capacity of a single SSD disk.

[0087] Specifically, since the bandwidth of the distributed object storage cluster mainly depends on the number and performance of HDD hard disks, the number of HDD hard disks can be calculated based on performance requirements.

[0088] In the embodiment of the present invention, the read and write bandwidth of the distributed object storage cluster is mainly determined by the read and write bandwidth and number of the HDD disks, and the read and write IOPS of the distributed object storage cluster is also mainly determined by the read and write IOPS and number of the HDD disks after testing. These two types of basic data can both obtain the limit values ​​in the test environment. It can be obtained that the average read bandwidth of each HDD hard disk is R bandwidth , the average write bandwidth is W bandwidth , under different IO granularity, the corresponding read IOPS is R iops , write IOPS is W iops .

[0089] In the embodiment of the present invention, the performance requirements of the user may include: a read / write bandwidth requirement value and a read / write IOPS requirement value.

[0090] Correspondingly, the number of HDD disks written = write bandwidth requirement ÷ W bandwidth , read HDD disk quantity = read bandwidth requirement ÷ R bandwidth In order to ensure the maximum performance, the maximum values ​​of the two quantities can be obtained as the number of HDD hard disks. In order to ensure the maximum performance, the maximum values ​​of the two quantities can be obtained as the number of HDD hard disks.

[0091] Write HDD disk quantity = write IOPS requirement ÷ W iops , Read HDD disk quantity = Read IOPS requirement ÷ R iops ,In order to ensure the maximum performance, the maximum value of the two quantities can be obtained as the number of HDD hard disks.

[0092] Further, the estimated number of HDD disks is determined based on the larger one of the number of HDD disks determined according to the read / write bandwidth requirement value and the number of HDD disks determined according to the read / write IOPS requirement value.

[0093] Furthermore, in the embodiment of the present invention, the types of servers are divided into capacity type, performance type and standard type. The number of SSD and HDD disks equipped on each type of server is different, resulting in different server prices. Therefore, in addition to correctly and reasonably estimating the number of servers according to actual needs, the final cost requirements must also be considered.

[0094] Therefore, in the embodiment of the present invention, the number of different types of servers required can be determined according to the number of SSD and HDD disks matched with different types of servers, as well as the calculated estimated number of SSD disks and estimated number of HDD disks, and then the total price can be obtained according to the number of different types of servers and the corresponding prices, and then the lowest cost solution can be determined, thereby determining the server type and the corresponding minimum number of servers. For example, a standard 12-slot server is selected, matched with 2 SSD disks + 10 HDDs, and the minimum number of servers = max (number of SSS disks ÷ 2, number of HDD disks ÷ 10).

[0095] In the embodiment of the present invention, the calculation of the data pool hard disk capacity first requires confirming the data protection method used in the user's data protection requirements, and then determining the redundancy ratio of the data pool of the distributed object storage cluster.

[0096] If the user does not specify data protection requirements, or does not specify the specific N+M number or number of copies of EC, the selection can be made based on the storage scenario and user storage requirements.

[0097] Specifically, step S106 includes the following two situations:

[0098] When it is determined in the data protection requirement to use EC erasure code for data protection: determine the number of data blocks N and the number of check blocks M; when the minimum number of servers is less than N+M, determine the estimated number of servers to be N+M+1; when the minimum number of servers is greater than or equal to N+M, determine the estimated number of servers to be the minimum number of servers.

[0099] When it is determined in the data protection requirement to adopt the replica mode for data protection: the estimated number of servers is determined to be the product of the minimum number of servers and the number of replicas.

[0100] In the embodiment of the present invention, the EC erasure code adopts a 4K alignment strategy and needs to meet the user's requirement to ensure the overall security of the system in the event of a failure. In addition, the EC erasure code also needs to meet the requirement that the number of storage servers is greater than N+M. According to the calculated minimum number of servers, the most cost-effective configuration among different EC ratios is selected.

[0101] For example, if the minimum number of servers calculated is 5 and the user's data protection needs determine the use of the replica mode, the closest and most cost-effective EC configuration ratio is 4+2. However, in order to meet the requirement that the number is greater than N+M, the minimum number of servers becomes 7. If the minimum number of servers calculated is 8, then the redundancy ratio of the EC configuration ratio of 4+2 is used without adjustment.

[0102] If the user determines to adopt the replica mode based on their data protection needs, the estimated number of servers = minimum number of servers × number of replicas.

[0103] In the embodiment of the present invention, after the estimated number of servers is determined based on the data protection solution, the disk capacity may be further determined.

[0104] Specifically, step S107 includes the following sub-steps:

[0105] S1071, determining the raw capacity of the data pool according to the storage capacity requirement, the number of data blocks N, the number of check blocks M, and a preset capacity usage alarm threshold.

[0106] S1072: Determine the single disk capacity of the HDD disk according to the raw capacity of the data pool, the estimated number of servers, and the number of HDD hard disks of a single server.

[0107] In the embodiment of the present invention, the raw capacity of the data pool = storage space capacity requirement × (N + M) ÷ N ÷ preset capacity usage alarm threshold, where the preset capacity usage alarm threshold may be 0.85.

[0108] The capacity of a single disk in the data pool = the raw capacity of the data pool / the minimum number of servers / the number of HDD hard disks on a single server. The capacity of a single HDD disk is generally 12T or 16T. You can choose the closest capacity based on the calculation result.

[0109] The capacity of a single SSD disk is generally 960G, 1.92T or 3.2T. If the cost can be controlled, you can also choose a high-performance nvme SSD to create an index pool.

[0110] After a series of calculations, the final estimated number of servers is determined to meet the overall capacity space requirements, the number of objects that can be stored, and the data protection requirements. However, in order to achieve the best performance, a matching number of application servers must also be selected. Under the conditions that the performance of the application server and the network bandwidth allow, the storage server can fully utilize the performance of the HDD disk and SSD disk to improve the utilization of the server.

[0111] In the embodiment of the present invention, the number of storage buckets of the distributed object storage cluster may be further determined.

[0112] In a distributed object storage cluster, buckets may receive a large number of read / write / delete operations at the same time. Without affecting performance, it is necessary to control the upper limit of the number of objects in each bucket at the business level to avoid overflow and cause performance jitter. Therefore, the number of buckets that need to be created = (number of large objects + number of small objects) ÷ maximum number of objects in a single bucket. When the number of objects written to the bucket is close to the threshold, flow control or flow switching can be performed to write the current main business to other buckets or write in a dispersed manner. Ensure that the upper-level business is not aware of it.

[0113] Based on the above embodiments, in the embodiments of the present invention, the estimated number of servers, the number of SSD and HDD disks equipped with each server, the capacity of a single HDD disk, the capacity of a single SSD disk, and the number of storage buckets can be determined, so that the configuration plan of the distributed object storage cluster that meets the user's needs can be determined through in-depth analysis of the user's different needs, such as performance requirements, capacity requirements, and data protection requirements.

[0114] In the embodiment of the present invention, the index pool space occupancy value, the DB space occupancy value, and the preset read bandwidth average, write bandwidth average, read IOPS average, and write IOPS average of the HDD disk are determined by the following steps:

[0115] S1, in the distributed object storage test cluster, uses SSD disks to create an index pool, and uses HDD disks to create a data pool with a preset redundancy ratio.

[0116] S2, in the test environment, upload a preset number of objects of preset size to the storage bucket through the client connection to the RGW gateway, collect the index pool usage capacity before uploading, the DB space already used by each OSD, and the capacity occupied by the index pool after uploading, and the DB space occupied by each OSD, and determine the index pool space occupancy value and DB space occupancy value of each object of the preset size.

[0117] S3, in the distributed object storage test cluster, obtain the read and write bandwidth and read and write IOPS of the HDD disk, and obtain the preset read bandwidth average, write bandwidth average, read IOPS average, and write IOPS average of the HDD disk.

[0118] S4, saving the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the object of the preset size.

[0119] In the embodiment of the present invention, considering that in distributed object storage, the object uploaded by the client to the storage cluster will generate three types of data, the index data will be stored in the index pool, the metadata information will be stored in the DB space allocated by each OSD, and the data will be stored in the data pool. Therefore, the three types of space occupied by each object in the storage cluster can be calculated / prepared in advance.

[0120] Specifically, for ease of understanding, combined with Figure 3 The basic data determination step in the distributed object storage demand assessment method proposed in an embodiment of the present invention is explained.

[0121] In the embodiment of the present invention, in the distributed object storage test cluster, an SSD disk is used to create a three-copy data security index pool. Due to different usage scenarios and data protection requirements, the redundancy ratios used in the data pool may be different. Therefore, on the test cluster, multiple data pools with different redundancy ratios can be created for testing. Thus, the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the object obtained under different redundancy ratios are obtained.

[0122] During the test, the client uploads objects to the storage bucket by connecting to the RGW gateway. In order to better and more accurately obtain the actual space occupied by each object, if the available capacity of the cluster allows, objects can be uploaded in multiple times, and at least 10 million objects of the same size can be uploaded to the cluster each time. Take uploading 10 million 1MB objects to a cluster with a redundancy ratio of EC 4+2 as an example:

[0123] Before uploading, first record the capacity used by the index pool, the capacity used by the data pool, and the DB space occupied by each OSD in the cluster. After uploading, record the capacity occupied by the index pool, the capacity occupied by the data pool, and the capacity occupied by each OSD again, calculate the difference between the three types of space, and then calculate the three basic data results as follows:

[0124] The space occupied by each object in the index pool = index pool capacity increment ÷ 10 million

[0125] The space occupied by each object in the data pool = data pool capacity increment ÷ 10 million

[0126] The space occupied by each object in the DB = the sum of all OSD DB increments ÷ the number of rados objects ÷ 6

[0127] According to the above calculation method, the actual space occupied by objects of sizes such as 4KB, 32KB, 256KB, 4MB, 10MB, 22MB, and 34MB can be calculated respectively.

[0128] After actual calculation, it is found that the index pool space occupied by objects of different sizes in the cluster is basically the same. The estimated fixed value V can be used for subsequent user demand conversion. index Make an estimate.

[0129] In addition, the space occupied by objects of different sizes in the DB is also kept at a fixed value V db The fixed value V can be used approximately when converting user requirements in the future. db Make an estimate.

[0130] In terms of performance, the read and write bandwidth of the cluster is mainly determined by the read and write bandwidth and number of HDD disks, and the read and write IOPS of the cluster is also mainly determined by the read and write IOPS and number of HDD disks after testing. In the test environment, both of these basic data can reach their limit values. Assume that the average read bandwidth of each HDD hard disk is R bandwidth, The average write bandwidth is W bandwidth , under different IO granularity, the corresponding read IOPS is R iops , write IOPS is Wiops .

[0131] As a result, several types of basic data required for demand conversion can be acquired in advance and archived, and do not need to be performed again in subsequent demand assessments.

[0132] S100, according to the redundancy ratio and the average object size, obtain the index pool space occupancy value, DB space occupancy value and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the HDD disk of the object of the corresponding size tested in the distributed object storage test cluster with the corresponding redundancy ratio.

[0133] In the embodiment of the present invention, step S100 may be performed after step S101 and before step S103.

[0134] In the embodiment of the present invention, in the actual application process, the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the HDD disk of the object of the corresponding size, which are pre-stored and tested in the distributed object storage test cluster with the corresponding redundancy ratio, can be obtained according to the redundancy ratio and the average object size in the user demand. In this way, the most accurate estimation result can be obtained in the subsequent demand conversion process.

[0135] In the embodiment of the present invention, in actual application, a unified fixed value may be directly used for calculation to simplify the estimation process.

[0136] Based on the same inventive concept, an embodiment of the present invention further provides a distributed object storage demand assessment device, the device comprising:

[0137] A demand decomposition module is used to obtain storage demand information, wherein the storage demand information includes: storage capacity demand, performance demand and data protection demand; the storage capacity demand represents the storage space of the distributed object storage cluster, the performance demand represents the read and write bandwidth information and read and write IOPS information of the distributed object storage cluster, and the data protection demand represents the redundancy ratio of the data pool of the distributed object storage cluster;

[0138] The demand conversion module is used to determine the number of objects that need to be stored according to the storage capacity requirement; determine the estimated number of SSD disks based on the number of objects, the predetermined index pool space occupancy value, the DB space occupancy value, and the capacity of a single SSD disk; determine the estimated number of HDD disks according to the performance requirements and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk; determine the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks; adjust the minimum number of servers based on the data protection requirements to obtain the estimated number of servers; determine the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers, and the number of HDD hard disks of a single server;

[0139] The index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

[0140] In an optional implementation of the embodiment of the present invention, the demand conversion module includes: an object quantity determination unit, which is used to:

[0141] In a case where the storage capacity requirement includes an average object size requirement, determining the number of objects to be stored according to the storage capacity requirement and the average object size requirement;

[0142] When the storage capacity requirement does not include the average object size requirement, the average object size is estimated according to the storage scenario corresponding to the storage requirement, and the number of objects to be stored is determined according to the storage capacity requirement and the estimated average object size.

[0143] In an optional implementation of the embodiment of the present invention, the demand conversion module includes: a server quantity adjustment unit, which is used to:

[0144] In the case where it is determined in the data protection requirement that EC erasure code is used for data protection;

[0145] Determine the number of data blocks N and the number of check blocks M;

[0146] When the minimum number of servers is less than N+M, the estimated number of servers is determined to be N+M+1;

[0147] When the minimum number of servers is greater than or equal to N+M, the estimated number of servers is determined to be the minimum number of servers.

[0148] In an optional implementation manner of the embodiment of the present invention, the server quantity adjustment unit is further used to:

[0149] In the case where it is determined in the data protection requirement that a replica mode is used for data protection;

[0150] The estimated number of servers is determined as the product of the minimum number of servers and the number of replicas.

[0151] In an optional implementation of the embodiment of the present invention, the demand conversion module includes: a capacity determination unit, which is used to:

[0152] Determine the raw capacity of the data pool according to the storage capacity requirement, the number of data blocks N, the number of check blocks M and the preset capacity usage alarm threshold;

[0153] The single disk capacity of the HDD disk is determined according to the raw capacity of the data pool, the estimated number of servers, and the number of HDD hard disks of a single server.

[0154] In an optional implementation of the embodiment of the present invention, the index pool space occupancy value, the DB space occupancy value, and the preset read bandwidth average, write bandwidth average, read IOPS average, and write IOPS average of the HDD disk are determined by the following steps:

[0155] In the distributed object storage test cluster, use SSD disks to create an index pool, and use HDD disks to create a data pool with a preset redundancy ratio;

[0156] In the test environment, upload a preset number of objects of preset size to the storage bucket through the client connection to the RGW gateway, collect the index pool usage capacity before uploading, the DB space used by each OSD, and the capacity occupied by the index pool and the DB space occupied by each OSD after uploading, and determine the index pool space occupancy value and DB space occupancy value of each object of the preset size;

[0157] In the distributed object storage test cluster, the read and write bandwidth and read and write IOPS of the HDD disk are obtained, and the preset read bandwidth average, write bandwidth average, read IOPS average and write IOPS average of the HDD disk are obtained;

[0158] Save the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the object of the preset size;

[0159] The device also includes: a basic data acquisition unit, which is used to: acquire, according to the redundancy ratio and the average object size, the index pool space occupancy value, the DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of objects of corresponding sizes tested in a distributed object storage test cluster with a corresponding redundancy ratio, which are stored.

[0160] Specifically, Figure 4 The structure diagram of the distributed object storage demand assessment device provided by the embodiment of the present invention is shown as follows: Figure 4 As shown, in an embodiment of the present invention, the basic data acquisition module can respectively obtain the space occupied by the object in the index pool, the DB space occupied by the underlying rados objects, the standard read and write bandwidth and IOPS of the HDD disk at different IO granularities (the average read bandwidth, average write bandwidth, average read IOPS and average write IOPS of the HDD disk), and the upper limit of the number of objects that can be stored in the bucket from the pre-calculated and saved basic data.

[0161] Among them, the space occupied by objects in the index pool, the DB space occupied by the underlying rados objects, the standard read and write bandwidth and IOPS of the HDD disk at different IO granularities are used to determine the optimal configuration plan for the storage server, and the upper limit of the number of objects that a bucket can store is used to determine the minimum number of storage buckets in the storage cluster.

[0162] In an embodiment of the present invention, the demand decomposition module can analyze the storage capacity requirements in the storage requirements input by the user to determine whether it contains a demand description of capacity and object size, and then the demand conversion module calculates the number of objects that need to be stored based on the demand description, and calculates the number of SSD disks based on the basic data obtained by the basic data acquisition module.

[0163] The demand decomposition module can also analyze the performance requirements in the storage requirements input by the user to determine whether the read / write bandwidth and IOPS requirements are included, and then the demand conversion module calculates the number of HDD disks based on the read / write bandwidth and IOPS requirements and the basic data obtained by the basic data acquisition module. The demand decomposition module can also analyze the data protection requirements in the storage requirements input by the user, and then the demand conversion module can determine the optimal configuration plan for the storage server based on the calculated number of SSD disks and HDD disks combined with the data protection requirements.

[0164] Based on the above embodiments, a parameterized demand model can be constructed, and a deep parameterized decomposition of customer needs can be achieved based on a demand decomposition module to design a configuration solution that better meets customer needs, thereby improving demand matching.

[0165] Based on the above embodiments, comprehensive performance analysis and cost-benefit analysis can be integrated, and accurate demand analysis and multi-dimensional optimization design can help promote efficient use of server resources and reduce resource waste, which is of great significance for reducing operating costs and improving system efficiency.

[0166] Based on the above embodiments, the customization capability of the systematic distributed object storage cluster can be realized, which greatly enhances the customized service capability of the distributed object storage system. In addition, the future scalability and maintainability of the system are taken into consideration to ensure that the configuration solution can easily adapt to future technology upgrades or changes in demand.

[0167] In summary, the technical solution provided by the embodiment of the present invention can significantly improve the utilization and cost-effectiveness of storage resources and save server operating costs by accurately matching user needs and optimizing configuration solutions. It can also customize storage services according to the different needs of different customers. It is especially practical for enterprises of different industries and sizes and can meet diverse and specific storage needs.

[0168] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the distributed object storage demand assessment method as described in any of the above embodiments are implemented.

[0169] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the distributed object storage demand assessment method described in any of the above embodiments are implemented.

[0170] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the distributed object storage demand assessment method described in any of the above embodiments.

[0171] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0172] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0173] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0177] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0178] The above is a detailed introduction to a distributed object storage demand assessment method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A distributed object storage demand assessment method, characterized in that: The method comprises: Acquire storage demand information, the storage demand information including: storage capacity demand, performance demand and data protection demand; the storage capacity demand represents the storage space of the distributed object storage cluster, the performance demand represents the read and write bandwidth information and read and write IOPS information of the distributed object storage cluster, and the data protection demand represents the redundancy ratio of the data pool of the distributed object storage cluster; Determine the number of objects that need to be stored according to the storage capacity requirement; Determine the estimated number of SSD disks based on the number of objects, the predetermined index pool space occupancy value, the DB space occupancy value, and the capacity of a single SSD disk; Determine the estimated number of HDD disks based on performance requirements and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk; Determine the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks; Based on the data protection requirements, the minimum number of servers is adjusted to obtain an estimated number of servers; Determine the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers and the number of HDD hard disks of a single server; The index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

2. The distributed object storage demand assessment method according to claim 1, characterized in that: According to the storage capacity requirement, determine the number of objects that need to be stored, including: In a case where the storage capacity requirement includes an average object size requirement, determining the number of objects to be stored according to the storage capacity requirement and the average object size requirement; When the storage capacity requirement does not include the average object size requirement, the average object size is estimated according to the storage scenario corresponding to the storage requirement, and the number of objects to be stored is determined according to the storage capacity requirement and the estimated average object size.

3. The distributed object storage demand assessment method according to claim 1, characterized in that: Based on the data protection requirements, the minimum number of servers is adjusted to obtain an estimated number of servers, including: In the case where it is determined in the data protection requirement that EC erasure code is used for data protection; Determine the number of data blocks N and the number of check blocks M; When the minimum number of servers is less than N+M, the estimated number of servers is determined to be N+M+1; When the minimum number of servers is greater than or equal to N+M, the estimated number of servers is determined to be the minimum number of servers.

4. The distributed object storage demand assessment method according to claim 1, characterized in that: Based on the data protection requirements, the minimum number of servers is adjusted to obtain an estimated number of servers, including: In the case where it is determined in the data protection requirement that a replica mode is used for data protection; The estimated number of servers is determined as the product of the minimum number of servers and the number of replicas.

5. The distributed object storage demand assessment method according to claim 3, characterized in that: Determine the single disk capacity of the HDD disk based on the storage capacity requirements, the estimated number of servers, and the number of HDD hard disks on a single server, including: Determine the raw capacity of the data pool according to the storage capacity requirement, the number of data blocks N, the number of check blocks M and the preset capacity usage alarm threshold; The single disk capacity of the HDD disk is determined according to the raw capacity of the data pool, the estimated number of servers, and the number of HDD hard disks of a single server.

6. The distributed object storage demand assessment method according to any one of claims 1 to 5, characterized in that: The index pool space occupancy value, DB space occupancy value, and the preset read bandwidth average, write bandwidth average, read IOPS average, and write IOPS average of the HDD disk are determined by the following steps: In the distributed object storage test cluster, use SSD disks to create an index pool, and use HDD disks to create a data pool with a preset redundancy ratio; In the test environment, upload a preset number of objects of preset size to the storage bucket through the client connection to the RGW gateway, collect the index pool usage capacity before uploading, the DB space used by each OSD, and the capacity occupied by the index pool and the DB space occupied by each OSD after uploading, and determine the index pool space occupancy value and DB space occupancy value of each object of the preset size; In the distributed object storage test cluster, the read and write bandwidth and read and write IOPS of the HDD disk are obtained, and the preset read bandwidth average, write bandwidth average, read IOPS average and write IOPS average of the HDD disk are obtained; Save the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of the object of the preset size; The method further comprises: According to the redundancy ratio and the average object size, the index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, and average read IOPS and average write IOPS of objects of corresponding sizes obtained by testing in the distributed object storage test cluster with the corresponding redundancy ratio are obtained.

7. A distributed object storage demand assessment device, characterized in that: The device comprises: A demand decomposition module is used to obtain storage demand information, wherein the storage demand information includes: storage capacity demand, performance demand and data protection demand; the storage capacity demand represents the storage space of the distributed object storage cluster, the performance demand represents the read and write bandwidth information and read and write IOPS information of the distributed object storage cluster, and the data protection demand represents the redundancy ratio of the data pool of the distributed object storage cluster; The demand conversion module is used to determine the number of objects that need to be stored according to the storage capacity requirement; determine the estimated number of SSD disks based on the number of objects, the predetermined index pool space occupancy value, the DB space occupancy value, and the capacity of a single SSD disk; determine the estimated number of HDD disks according to the performance requirements and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk; determine the minimum number of servers based on the estimated number of SSD disks and the estimated number of HDD disks; adjust the minimum number of servers based on the data protection requirements to obtain the estimated number of servers; determine the single disk capacity of the HDD disk according to the storage capacity requirement, the estimated number of servers, and the number of HDD hard disks of a single server; The index pool space occupancy value, DB space occupancy value, and the average read bandwidth, average write bandwidth, average read IOPS, and average write IOPS of each HDD disk are obtained by testing on a distributed object storage test cluster.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the distributed object storage demand assessment method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the distributed object storage demand assessment method described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the distributed object storage demand assessment method described in any one of claims 1-6 are implemented.

Images