Heterogeneous disk-based distributed storage optimization method, system, device and medium

Abstract

The application discloses a distributed storage optimization method, system, device and medium based on heterogeneous disks. The method performs performance benchmarking on all types of disks according to first disks and second disks to obtain the write throughput and read throughput of each disk. The overall cost of each disk combination is calculated. Based on the overall cost, the write throughput and the read throughput, the first cost performance of each disk combination under a known workload is calculated, and the second cost performance of each disk combination under an unknown workload or a changing workload is calculated. According to the first cost performance of each disk combination, a first target disk configuration is determined, and according to the second cost performance of each disk combination, a second target disk configuration is determined. According to the first target disk configuration or the second target disk configuration, data distributed storage is performed. The application can eliminate I / O path competition and reduce disk usage cost while ensuring disk read-write throughput.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a distributed storage optimization method, system, device and medium based on heterogeneous disks. BACKGROUND

[0002] With the rapid development of big data and cloud computing, the amount of data generated by modern applications is growing exponentially, which poses unprecedented challenges to the write performance, scalability and reliability of storage systems. The traditional B+ tree storage structure has a sharp decline in performance when facing a large number of random write operations, and it is difficult to meet the needs of modern applications.

[0003] Under this background, the Log-Structured Merge-tree (LSM-tree) architecture is widely adopted due to its excellent write performance. LSM-tree significantly improves write throughput by converting random writes to sequential writes, and becomes the preferred architecture for modern key-value storage systems. This architecture has been widely used in NoSQL databases, big data processing platforms and real-time data analysis systems.

[0004] However, the existing distributed storage method based on LSM-tree has a serious performance bottleneck caused by I / O path competition, and the existing method cannot balance the relationship between disk throughput and disk usage cost, that is, when the throughput is high, the cost is also high, and when the cost is low, the throughput is also low. SUMMARY

[0005] The present application aims to provide a distributed storage optimization method, system, device and medium based on heterogeneous disks, which can eliminate I / O path competition and reduce disk usage cost while ensuring disk read and write throughput.

[0006] In a first aspect, the embodiments of the present application provide a distributed storage optimization method based on heterogeneous disks, which comprises:

[0007] allocating a first disk for Raft log persistence and pre-write log write operations, and allocating a second disk for background compression read and write operations;

[0008] According to the first disk and the second disk, performance benchmarking is performed on all types of disks to obtain the write throughput and read throughput of each disk;

[0009] According to the disk capacity and unit capacity price of each type of disk, the overall cost of each disk combination is calculated;

[0010] Based on the overall cost, the write throughput and the read throughput, the first cost-effectiveness of each disk combination under known workloads is calculated;

[0011] based on the total cost, the write throughput and the read throughput, calculating a second cost performance of each disk combination under an unknown workload or a workload change;

[0012] determining a first target disk configuration according to the first cost performance of each disk combination, and determining a second target disk configuration according to the second cost performance of each disk combination;

[0013] performing data distributed storage according to the first target disk configuration or the second target disk configuration.

[0014] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0015] The method allocates a first disk for Raft log persistence and pre-write log write operation, and allocates a second disk for background compression read-write operation; according to the first disk and the second disk, performance benchmarking is performed on all types of disks to obtain the write throughput and the read throughput of each disk; according to the disk capacity and the unit capacity price of each type of disk, the total cost of each disk combination is calculated; based on the total cost, the write throughput and the read throughput, the first cost performance of each disk combination under a known workload is calculated; based on the total cost, the write throughput and the read throughput, the second cost performance of each disk combination under an unknown workload or a workload change is calculated; a first target disk configuration is determined according to the first cost performance of each disk combination, and a second target disk configuration is determined according to the second cost performance of each disk combination; data distributed storage is performed according to the first target disk configuration or the second target disk configuration. In this way, through the I / O path physical separation scheme of the dual-disk architecture, resource competition between I / O operation types can be eliminated, and the write path and the compression path are completely decoupled; by comprehensively considering the total cost, the write throughput and the read throughput, the performance of each disk combination under a known workload, an unknown workload or a workload change is calculated, which can reduce the disk usage cost while ensuring the disk read-write throughput, and systematically solves the storage configuration optimization problem in the cloud environment.

[0016] In some embodiments, the first disk is allocated for Raft log persistence and pre-write log write operation, including:

[0017] obtaining a write request sent by a client, and encapsulating the write request as a Raft log entry;

[0018] copying the Raft log entry to a follower node;

[0019] persisting the Raft log to the first disk;

[0020] When a preset number of follower nodes are confirmed, the leader performs an apply operation to write data into a prewrite log;

[0021] persist the prewrite log to the first disk.

[0022] In some embodiments, the performance benchmarking of all types of disks according to the first disk and the second disk obtains the write throughput and the read throughput of each disk, including:

[0023] The performance benchmarking of all types of disks as the first disk obtains the write throughput of each disk;

[0024] The performance benchmarking of all types of disks as the second disk obtains the read throughput of each disk.

[0025] In some embodiments, the calculation of the overall cost of each disk combination according to the disk capacity and the unit capacity price of each type of disk includes:

[0026] Obtain the server instance fixed cost;

[0027] Multiply the disk capacity and the unit capacity price of each type of disk to obtain the disk cost corresponding to each type of disk;

[0028] Add the disk cost corresponding to each type of disk in each disk combination to obtain the cost addition result corresponding to each disk combination;

[0029] Add the cost addition result corresponding to each disk combination and the server instance fixed cost to obtain the overall cost of each disk combination.

[0030] In some embodiments, the calculation of the first cost performance of each disk combination under the known workload based on the overall cost, the write throughput and the read throughput includes:

[0031] Obtain the proportion of read operations in the workload and the proportion of write operations in the workload;

[0032] Calculate the proportion of read operations and the read throughput of each disk to obtain a first ratio result;

[0033] Calculate the proportion of write operations and the write throughput of each disk to obtain a second ratio result;

[0034] According to the first ratio result, the second ratio result and the overall cost corresponding to each disk combination, calculate the first cost performance of each disk combination under the known workload.

[0035] In some embodiments, the calculating the second cost-effectiveness of each disk combination under unknown workloads or workload changes based on the overall cost, the write throughput and the read throughput comprises:

[0036] ;

[0037] wherein, represents the second cost-effectiveness, represents the overall cost of the configuration , represents the first class of disks as the first disk, represents the first class of disks as the second disk, represents the first class of disks as the second disk, represents the read throughput of the first class of disks, represents the write throughput of the first class of disks, represents the write throughput of the first class of disks, represents the natural logarithm. In some embodiments, the determining the first target disk configuration according to the respective first cost-effectiveness of all disk combinations and the second target disk configuration according to the respective second cost-effectiveness of all disk combinations comprises:

[0038] selecting, as the first target disk configuration, the disk combination corresponding to the maximum first added ratio from the respective first cost-effectiveness of all disk combinations;

[0039] selecting, as the second target disk configuration, the disk combination corresponding to the maximum second added ratio from the respective second cost-effectiveness of all disk combinations.

[0040] selecting, as the second target disk configuration, the disk combination corresponding to the maximum second added ratio from the respective second cost-effectiveness of all disk combinations.

[0041] In a second aspect, the embodiments of the present application further provide a distributed storage optimization system based on heterogeneous disks, which comprises:

[0042] a heterogeneous disk allocation unit configured to allocate a first disk for Raft log persistence and pre-write log write operation, and allocate a second disk for background compression read-write operation;

[0043] a performance benchmarking unit configured to perform performance benchmarking on all types of disks according to the first disk and the second disk to obtain the write throughput and the read throughput of each disk;

[0044] an overall cost calculation unit configured to calculate the overall cost of each disk combination according to the disk capacity and the unit capacity price of each type of disk;

[0045] ​a first cost-effectiveness calculation unit configured to calculate a first cost-effectiveness of each disk combination under a known workload condition based on the total cost, the write throughput and the read throughput;

[0046] a second cost-effectiveness calculation unit configured to calculate a second cost-effectiveness of each disk combination under an unknown workload or workload change condition based on the total cost, the write throughput and the read throughput;

[0047] a disk configuration determination unit configured to determine a first target disk configuration according to the first cost-effectiveness of each disk combination, and determine a second target disk configuration according to the second cost-effectiveness of each disk combination;

[0048] a distributed storage unit configured to perform data distributed storage according to the first target disk configuration or the second target disk configuration.

[0049] In a third aspect, an electronic device is provided, which includes at least one control processor and a memory connected to the at least one control processor in communication; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the above-mentioned heterogeneous disk-based distributed storage optimization method.

[0050] In a fourth aspect, a computer readable storage medium is provided, which stores computer executable instructions for causing a computer to perform the above-mentioned heterogeneous disk-based distributed storage optimization method.

[0051] It can be understood that the above-mentioned second aspect to fourth aspect have the same beneficial effects as the first aspect compared with the related art, and the related description can be referred to the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0052] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0053] Figure 1 is a flowchart of an embodiment of the heterogeneous disk-based distributed storage optimization method provided by the present application;

[0054] Figure 2 is an experimental result diagram of the compression on the write performance in the best embodiment of the heterogeneous disk-based distributed storage optimization method provided by the present application;

[0055] Figure 3Fig. 1 is a schematic diagram of a dual-disk structure of a distributed storage system based on Raft and LSM tree in an embodiment of the distributed storage optimization system based on heterogeneous disks provided in the present application;

[0056] Figure 4 Fig. 1 is a schematic diagram of a dual-disk structure of a distributed storage system based on Raft and LSM tree in an embodiment of the distributed storage optimization system based on heterogeneous disks provided in the present application;

[0057] Figure 5 Fig. 1 is a schematic diagram of a dual-disk structure of a distributed storage system based on Raft and LSM tree in an embodiment of the distributed storage optimization system based on heterogeneous disks provided in the present application; DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below with reference to the accompanying drawings. Examples of the embodiments of the present application are shown in the drawings, in which the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0059] In the description of the present application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features or the order of the indicated technical features.

[0060] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc., is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present application.

[0061] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0062] First, some terms involved in the present application are analyzed:

[0063] Log-structured merge tree (LSM-tree): a write-optimized data structure that converts random writes into sequential writes to improve write performance. Data is first written to memory and then flushed to disk in batches.

[0064] Raft / Paxos consensus algorithm: a distributed consensus protocol used to ensure data consistency in distributed systems by electing a leader node to coordinate log replication.

[0065] Write-Ahead Log (WAL): A database technique that records operations to a log before modifying data, ensuring data durability.

[0066] Sorted String Table (SSTable): An ordered data file stored on disk in LSM-tree.

[0067] MemTable: A memory data structure in LSM-tree used to temporarily store newly written data.

[0068] Compaction: A background maintenance operation of LSM-tree that combines and organizes multiple layers of SSTable files, removes expired data, and reduces storage space.

[0069] I / O Path Separation: Allocating different types of disk read and write operations to different physical storage devices to avoid resource contention.

[0070] Cost-Performance Ratio (CPR): A measure of the performance obtained per unit cost.

[0071] IOPS: Input / Output Operations Per Second, a measure of storage device performance.

[0072] Service Level Objective (SLO): A quantitative measure of system performance requirements.

[0073] Heterogeneous Disks: Multiple storage devices with different performance and price characteristics, such as Hard Disk Drive (HDD), Solid State Drive (SSD), and NVMe SSD.

[0074] Dual Disk Architecture: The storage architecture proposed in this application, using two independent disks to handle different types of I / O operations.

[0075] Write Amplification: The phenomenon where the actual amount of data written to storage media by the system is greater than the amount of data requested to be written by the user.

[0076] ATLAS (Adaptive Two-tier Log-Data Architecture): The cost-performance ratio model proposed in this application, which can give the best cost-performance ratio disk configuration under known and unknown workload characteristics.

[0077] YCSB (Yahoo! Cloud Serving Benchmark): a commonly used cloud storage and database system performance evaluation benchmark tool, used to simulate a variety of typical read-write workloads.

[0078] The existing distributed storage method based on LSM-tree has a serious performance bottleneck caused by I / O path competition, and the existing method cannot balance the relationship between disk throughput and disk use cost, that is, when the throughput is high, the cost is also high, and when the cost is low, the throughput is also low.

[0079] To solve the problems existing in the prior art, the application provides a distributed storage optimization method, system, device and medium based on heterogeneous disks.

[0080] Referring to Figure 1 The flowchart of the distributed storage optimization method based on heterogeneous disks provided by the embodiments of the application. The distributed storage optimization method based on heterogeneous disks is applied to an electronic device, which can be a server or a mobile terminal, etc. As Figure 1 shown, the distributed storage optimization method based on heterogeneous disks can include the following steps:

[0081] Step S101, allocating a first disk for Raft log persistence and pre-write log write operation, and allocating a second disk for background compression read-write operation;

[0082] Step S102, according to the first disk and the second disk, performing performance benchmarking on all types of disks to obtain the write throughput and read throughput of each disk;

[0083] Step S103, calculating the overall cost of each disk combination according to the disk capacity and unit capacity price of each type of disk;

[0084] Step S104, based on the overall cost, write throughput and read throughput, calculating the first cost performance of each disk combination under known workloads;

[0085] Step S105, based on the overall cost, write throughput and read throughput, calculating the second cost performance of each disk combination under unknown workloads or workload changes;

[0086] Step S106, determining a first target disk configuration according to the first cost performance of each disk combination, and determining a second target disk configuration according to the second cost performance of each disk combination;

[0087] Step S107, performing data distributed storage according to the first target disk configuration or the second target disk configuration.

[0088] In the embodiment, the first disk is allocated for Raft log persistence and prewrite log write operation, and the second disk is allocated for background compression read-write operation; performance benchmark tests are performed on all types of disks according to the first disk and the second disk, to obtain the write throughput and the read throughput of each disk; the overall cost of each disk combination is calculated according to the disk capacity and the unit capacity price of each type of disk; the first cost performance of each disk combination under the known workload is calculated based on the overall cost, the write throughput and the read throughput; the second cost performance of each disk combination under unknown workload or workload change is calculated based on the overall cost, the write throughput and the read throughput; the first target disk configuration is determined according to the first cost performance of each disk combination, and the second target disk configuration is determined according to the second cost performance of each disk combination; and data distributed storage is performed according to the first target disk configuration or the second target disk configuration. In this way, the resource competition between I / O operation types can be eliminated by the I / O path physical separation scheme of the dual-disk architecture, and the write path and the compression path are completely decoupled; by comprehensively considering the overall cost, the write throughput and the read throughput, the performance of each disk combination under known workload, unknown workload or workload change is calculated, which can reduce the disk usage cost while ensuring the disk read-write throughput, and systematically solve the storage configuration optimization problem in the cloud environment.

[0089] The first disk and the second disk described above can be one of the disks in the prior art, for example, a mechanical hard disk (HDD), a solid state disk (SSD), and a high-performance solid state disk (NVMe SSD). The first disk and the second disk can be the same or different.

[0090] The performance benchmark test on all types of disks described above can be a throughput test on all types of disks.

[0091] The disk capacity and the unit capacity price of each type of disk described above can be the disk capacity and the unit capacity price of each type of disk artificially set.

[0092] The data distributed storage according to the first target disk configuration or the second target disk configuration described above can be that the Raft log persistence, the prewrite log write operation and the background compression read-write operation are allocated to the first target disk configuration or the second target disk configuration.

[0093] In some embodiments, the first disk is allocated for Raft log persistence and prewrite log write operation, including:

[0094] The write request sent by the client is obtained, and the write request is encapsulated as a Raft log entry.

[0095] copying the Raft log entry to the follower node;

[0096] persisting the Raft log to the first disk;

[0097] after a preset number of follower nodes are confirmed, the leader performs an apply operation to write data to the write-ahead log;

[0098] persisting the write-ahead log to the first disk.

[0099] In the embodiment, the first disk is allocated for the Raft log persistence and the write operation of the write-ahead log, and a different disk is used for the background compression read-write operation, so that resource competition between I / O operation types is eliminated, and the write path and the compression path are completely decoupled.

[0100] The copying of the Raft log entry to the follower node can be copying the Raft log entry to the follower node through a network.

[0101] The persisting of the Raft log to the first disk can ensure persistence.

[0102] The preset number can be a number set by a person, and can be adjusted according to actual conditions, and the embodiment is not specifically limited.

[0103] In some embodiments, according to the first disk and the second disk, performance benchmarking is performed on all types of disks to obtain the write throughput and the read throughput of each disk, including:

[0104] The performance benchmarking is performed on all types of disks as the first disk to obtain the write throughput of each disk;

[0105] The performance benchmarking is performed on all types of disks as the second disk to obtain the read throughput of each disk.

[0106] In the embodiment, by performing the performance benchmarking on all types of disks according to the first disk and the second disk, the write throughput and the read throughput of each disk are obtained, which can lay a good data foundation for more accurate calculation of the performance of each disk combination in the later stage.

[0107] In some embodiments, according to the disk capacity and the unit capacity price of each type of disk, the total cost of each disk combination is calculated, including:

[0108] obtaining a server instance fixed cost;

[0109] multiplying the disk capacity and the unit capacity price of each type of disk to obtain the disk cost corresponding to each type of disk;

[0110] Add the disk cost of each type of disk in each disk combination to obtain a cost addition result corresponding to each disk combination;

[0111] Add the cost addition result corresponding to each disk combination to the server instance fixed cost to obtain the overall cost of each disk combination.

[0112] In the embodiment, the overall cost of each disk combination is calculated according to the disk capacity and the unit capacity price of each type of disk, which can lay a good data foundation for selecting an optimal disk combination for distributed storage in the later stage, so as to reduce the disk use cost while ensuring the disk read-write throughput.

[0113] The server instance fixed cost can be a server instance fixed cost artificially set by the user.

[0114] In some embodiments, based on the overall cost, the write throughput and the read throughput, a first cost performance of each disk combination under a known workload is calculated, including:

[0115] Obtaining the proportion of read operations in the workload and the proportion of write operations in the workload;

[0116] Calculating the proportion of read operations and the read throughput of each disk by ratio to obtain a first ratio result;

[0117] Calculating the proportion of write operations and the write throughput of each disk by ratio to obtain a second ratio result;

[0118] According to the first ratio result, the second ratio result and the overall cost corresponding to each disk combination, the first cost performance of each disk combination under the known workload is calculated.

[0119] In the embodiment, by calculating the first cost performance of each disk combination under the known workload based on the overall cost, the write throughput and the read throughput, an optimal disk combination in each disk combination under the known workload can be selected to lay a good data foundation for distributed storage, so as to reduce the disk use cost while ensuring the disk read-write throughput.

[0120] In some embodiments, based on the overall cost, the write throughput and the read throughput, a second cost performance of each disk combination under an unknown workload or a workload change is calculated, including:

[0121] ;

[0122] wherein, represents the second cost performance, represents the overall cost of the configuration ; represents the first represents the first disk, represents the first represents the second disk, represents the first represents the read throughput of the first disk, represents the first represents the write throughput of the first disk, represents the natural logarithm.

[0123] In the embodiment, by calculating the second cost performance of each disk combination under unknown workloads or workload changes based on the overall cost, write throughput and read throughput, the optimal disk combination in each disk combination under unknown workloads or workload changes can be selected for distributed storage, so as to lay a good data foundation, thereby reducing the disk use cost while ensuring the disk read-write throughput.

[0124] In some embodiments, the first target disk configuration is determined according to the respective first cost performance of all disk combinations, and the second target disk configuration is determined according to the respective second cost performance of all disk combinations, including:

[0125] The disk combination corresponding to the maximum first added ratio is selected from the respective first cost performance of all disk combinations as the first target disk configuration;

[0126] The disk combination corresponding to the maximum second added ratio is selected from the respective second cost performance of all disk combinations as the second target disk configuration.

[0127] In the embodiment, the disk combination corresponding to the maximum first added ratio is selected from the respective first cost performance of all disk combinations as the first target disk configuration, and the disk combination corresponding to the maximum second added ratio is selected from the respective second cost performance of all disk combinations as the second target disk configuration, which can reduce the disk use cost while ensuring the disk read-write throughput, and systematically solve the storage configuration optimization problem in the cloud environment.

[0128] For the convenience of those skilled in the art, a set of best embodiments is provided as follows:

[0129] With the rapid development of big data and cloud computing, the amount of data generated by modern applications grows exponentially, which poses unprecedented challenges to the write performance, scalability and reliability of storage systems. The traditional B+ tree storage structure has a sharp decline in performance when facing a large number of random write operations, and it is difficult to meet the needs of modern applications.

[0130] In this context, the Log-Structured Merge-tree (LSM-tree) architecture has been widely adopted for its superior write performance. By converting random writes into sequential writes, LSM-tree significantly improves write throughput and becomes the preferred architecture for modern key-value storage systems. This architecture has been widely applied in NoSQL databases, big data processing platforms, and real-time data analysis systems. However, single-node storage systems have inherent limitations in reliability and availability. To meet the stringent requirements of modern applications for high availability, fault tolerance, and data consistency, distributed storage systems have emerged. These systems introduce consensus algorithms such as Raft or Paxos to maintain data replicas across multiple nodes, ensuring that the system can continue to operate even if some nodes fail.

[0131] In the cloud computing environment, the cost of storage infrastructure accounts for a significant proportion of operating costs. Cloud service providers adopt a tiered pricing model, with different storage types having significant differences in performance and cost. For example, on major cloud platforms such as AWS and Azure, the cost of premium SSD storage can be 2 to 4 times that of standard HDD storage, and high-performance NVMe SSDs can cost up to 10 times that of standard HDDs when considering IOPS and throughput configurations. This cost difference poses a fundamental challenge for distributed storage system operators, i.e., how to find the most cost-effective storage configuration while meeting throughput service level objectives (SLOs).

[0132] The prior art has the following disadvantages:

[0133] Disadvantage one: I / O path competition leads to serious performance bottlenecks. Existing Raft and LSM-tree-based distributed storage systems have serious I / O resource competition problems. In a typical deployment scenario, the following key operations compete for the same physical storage device: 1) Raft log append operation: sequential write operation of consensus protocol; 2) LSM-tree WAL write: persistence operation of write-ahead log; 3) MemTable flush: periodic writing of in-memory data to L0 level SSTable file; 4) compression operation: intensive read-modify-write cycle across multiple LSM-tree levels. When the compression operation is executed, it consumes a large amount of I / O bandwidth: reading multiple large SSTable files (possibly up to GB level data volume), writing the merged output file to disk, and performing synchronization operations, which directly interferes with the execution of write operations.

[0134] Disadvantage two: lack of systematic cost-performance optimization method. Storage configuration decision in cloud environment faces complex cost-performance trade-off problem, while existing technology lacks quantitative optimization model. 1) Large price difference: the cost difference of different storage types is significant, the cost of advanced SSD storage may be 2 to 4 times that of standard HDD, and the cost of high-performance NVMe SSD may reach 10 times that of standard HDD when considering IOPS and throughput; 2) Configuration selection is complex: even if the I / O competition is solved by using dual-disk architecture, there is still a configuration selection problem. For example, for write-intensive workloads, SSD and SSD provide the lowest latency but double the cost, SSD and HDD provide fast foreground operations and sufficient compression performance, HDD and SSD may cause a bottleneck in Raft log persistence, and HDD and HDD minimize cost but may violate SLO. The optimal choice depends on specific throughput requirements, cost budget constraints, workload intensity and data volume; 3) Insufficient handling of workload changes: existing solutions are difficult to handle scenarios where the workload is unknown or changing. Production systems often experience workload changes (different operation combinations at different times), unpredictable patterns (sudden traffic peaks or application behavior changes), and demand evolution (applications transitioning from read-intensive to write-intensive). The configuration optimized for one workload type may not perform well under another.

[0135] Disadvantage three: limitations of existing optimization solutions. 1) Limitations of storage separation solutions: some solutions mainly focus on hot and cold data separation, which can optimize long-term storage costs, but do not fundamentally solve the I / O competition problem in the write critical path. Some hybrid storage solutions rely on specific high-end hardware (such as non-volatile memory NVM), which can speed up compression or reduce write amplification, but the deployment cost is high, and the availability is limited in general cloud environments; 2) Limitations of device selection solutions: Selecta and other automated device selection systems can select the optimal device based on workload characteristics, but mainly consider performance from the performance perspective, lack of cost-effective quantitative evaluation, and cannot achieve optimal configuration under budget constraints; 3) Limitations of cost optimization research: existing cost-performance optimization research mainly focuses on general cloud service performance prediction and diagnosis, lacks specialized models for LSM-tree distributed storage architecture, and cannot accurately reflect the performance characteristics and mutual influence of different I / O paths such as consensus log, WAL and compression operations.

[0136] The purposes of the present embodiment include the following points:

[0137] Objective 1: Physically isolate I / O paths to eliminate resource contention. Addressing the core issue of I / O path contention in existing technologies, the primary objective of this embodiment is to design an I / O path separation architecture based on heterogeneous disks. Specific objectives include: 1) Physically isolating critical paths: Dedicated disks to latency-sensitive Raft log persistence and WAL write operations, and separate background compression read / write operations to another independent disk, completely eliminating I / O resource contention. 2) Improving write performance stability: The dual-disk architecture eliminates competition for disk resources between compression and write operations, thereby eliminating interference with critical write paths and enabling the system to maintain stable write performance even during intensive compression. 3) Maintaining system compatibility: Performance optimization is achieved through storage configuration adjustments without altering the core logic of the existing distributed storage system, ensuring the practicality and deployability of the solution.

[0138] Objective Two: Establish a quantitative cost-performance optimization model. Addressing the lack of systematic optimization methods for storage configuration selection in cloud environments, the second objective of this embodiment is to establish a complete cost-performance evaluation system: 1) Construct a predictive model: A performance prediction model is established based on the harmonic average principle, capable of passing limited benchmark tests (…). (Complexity) Predicting any combination of disk configurations ( (1) Performance under different workloads; 2) Quantitative cost-effectiveness: Establish a cost-performance ratio evaluation model that comprehensively considers disk price, server cost and system throughput to provide a quantitative basis for configuration selection; 3) Support for multi-scenario optimization: For known workload scenarios, provide the optimal configuration selection based on specific read-write ratios; for unknown or changing workload scenarios, provide robust and universal cost-performance metrics through integral calculation.

[0139] Objective 3: To implement practical configuration optimization strategies. The third objective of this embodiment is to provide optimization solutions that can be applied in real-world production environments: 1) Reduce evaluation complexity: Through mathematical modeling, the time complexity of configuration evaluation is reduced from... Reduce to This makes it possible to optimize the configuration of a large number of storage types in the cloud environment; 2) Adapt to the characteristics of the cloud environment: The input parameters required by the model (read and write performance benchmarks, disk pricing) can be obtained through standard testing and public pricing information, which is convenient for application in actual systems; 3) Provide decision-making process: Establish a complete decision-making workflow from benchmark testing, cost calculation and performance evaluation to configuration selection, providing system operators with actionable optimization guidance.

[0140] By achieving the above objectives, the embodiment aims to provide a systematic solution for LSM-tree-based distributed storage systems that can significantly improve performance and optimize cost-effectiveness, achieving the optimal configuration of storage infrastructure costs under the premise of meeting performance requirements. The embodiment method specifically includes the following contents:

[0141] I. Dual-disk architecture design.

[0142] Although LSM-tree optimizes write performance by converting random writes to sequential writes, this optimization transfers the cost to compression operations. Compression is a key mechanism for LSM-tree to maintain efficient data management. It regularly merges and sorts multi-level SSTable files to eliminate redundant data, reclaim storage space, and maintain read efficiency. However, compression involves a large number of disk I / O operations. It reads multiple SSTable files, merges and sorts data in memory, and then writes the results back to disk. These operations consume a large amount of disk bandwidth and IOPS. More seriously, in typical deployment scenarios, compression operations compete with the system's critical write paths for the same physical storage devices, which can cause a significant decline in write performance.

[0143] To verify the impact of background compression operations on these critical write paths, the embodiment tests the throughput and average latency under default settings and with a large enough Memtable (without triggering compression). The experimental results are shown in Figure 2 Compared to the default case, when the Memtable is adjusted to be large enough, the write throughput of solid-state drives (SSD) and hard disk drives (HDD) is improved by several times (SSD from about 18k operations / second to nearly 39k operations / second, HDD from about 15k operations / second to 32k operations / second). At the same time, the average write latency is also significantly reduced, indicating that compression does indeed affect overall write performance.

[0144] To solve this problem, the embodiment proposes an I / O path separation solution based on a dual-disk architecture. The core idea of the embodiment is to physically isolate the I / O path of compression operations from the critical write path: use one disk 1 (i.e., the first disk) exclusively for delay-sensitive Raft log persistence and WAL write operations, while offloading compression read-write operations and SSTable storage to another independent disk 2 (i.e., the second disk). With this dual-disk architecture, compression operations no longer compete with write operations for disk resources, thus eliminating interference with the critical write path and allowing the system to maintain stable write performance.

[0145] As shown in Figure 3As shown in the dual-disk architecture proposed in this embodiment, a typical write request undergoes the following procedures: (1) the client sends a write request to the leader node; (2) the leader encapsulates the request as a Raft log entry and replicates the log entry to the follower nodes through the network; (3) the Raft log is persisted to disk 1 to ensure durability; (4) after the majority of the follower nodes confirm, the log reaches consensus and is committed, and the leader performs the apply operation; (5) the data is written to the WAL (Write-Ahead Log) of the underlying LSM-tree storage engine, and the WAL is also persisted to disk 1; (6) the data is then written to the MemTable, and when the MemTable reaches the threshold, it is flushed to disk 2 to generate SSTable files. At the same time, the background compaction operation is also independently performed on disk 2, reading and merging multiple SSTable files. In the entire process, the Raft log persistence and WAL persistence, which are sensitive to latency, are completed on disk 1, while the data storage and compaction operations are completely performed on disk 2, achieving physical isolation of the I / O path. Experimental results show that on SSD and HDD, this architecture respectively achieves 2.2 times and 2.1 times improvement in write throughput.

[0146] II. Design of the cost-performance model ATLAS.

[0147] The dual-disk architecture eliminates direct I / O competition, allowing the system to maintain consistent write performance even during intensive compression. However, this approach presents new challenges: how to determine the optimal storage configuration under known and unknown workloads. The cost-performance model ATLAS is used to evaluate the cost-effectiveness of different disk configurations under specific workloads and provides a theoretical basis for configuration selection in known and unknown workload scenarios.

[0148] In the dual-disk separation architecture, the log disk (i.e., disk 1) mainly affects the write performance of the system, while the data disk (i.e., disk 2) mainly affects the read performance. This model can predict the performance of any disk combination under different workloads through a limited number of benchmark tests.

[0149] For known read-write ratio workloads, the ATLAS-CPR (ATLAS-Cost Performance Ratio, known workload) performs the following procedures:

[0150] Phase 1: Benchmarking. In a cloud environment, there are types of disk available, and this embodiment first performs two types of performance tests on each disk type. First, all disks are tested for throughput when they are used as log disks and data disks respectively, and each disk is tested twice, one disk is , the and of each disk are derived. All disks are run as log disks under pure write load to derive their write throughput . Then, all disks are run as data disks under pure read load to derive their read throughput . Through these two tests, the performance characteristics of the disks in the write path and the read path are extracted, respectively.

[0151] Stage 2: Cost Calculation. For all possible disk combinations , the overall cost of each disk combination is calculated:

[0152] (1);

[0153] where denotes the overall cost of the configuration , Cserveris the fixed cost of the server instance, is the capacity of the thclass of disk (log disk), is the capacity of the thclass of disk (data disk), is the unit capacity price of the thclass of disk, and is the unit capacity price of the thclass of disk. Stage 3: Cost-Performance Evaluation. For each configuration, its cost-performance indicator under a known read-write ratio

[0154] is calculated:

[0155] (2); where

[0156] denotes the cost-performance under the known workload (i.e., the first cost-performance), denotes the proportion of read operations in the workload, denotes the proportion of write operations in the workload, denotes the thclass of disk as the first disk, denotes the thclass of disk as the second disk, denotes the read throughput of the thclass of disk, and denotes the write throughput of the thclass of disk.represents the first Class disk write throughput.

[0157] The formula takes into account the weighted influence of read and write performance, reflecting the overall processing capacity of the system under unit cost.

[0158] Phase 4: Configuration Selection. Among all candidate configurations, the most cost-effective solution is selected:

[0159] (3);

[0160] where, represents the optimal disk combination under known workload conditions (i.e., the first target disk configuration).

[0161] Phase 5: Deployment. The selected optimal configuration (i.e., the first target disk configuration) is deployed to the system for actual operation of the target workload.

[0162] For unknown or dynamically changing workload scenarios, ATLAS-UCPR (ATLAS-Universal Cost-Performance, Unknown Workload) calculates the average performance of all possible loads by integration, thereby selecting the most robust disk combination. The specific execution process is as follows:

[0163] Phase 1: Performance Benchmarking. The same as in ATLAS-CPR, performance tests are conducted on all types of disks, recording the write performance of each disk as a log disk ; record the read performance of each disk as a data disk .

[0164] Phase 2: Cost Calculation. Calculate the total cost for all disk combinations :

[0165] (4);

[0166] Phase 3: Universal Cost-Performance Evaluation. Since the workload is unknown, the CPR for all possible read-write ratios needs to be integrated to find the average:

[0167] (5);

[0168] Its closed-form solution is:

[0169] (6);

[0170] wherein, represents the general cost-performance ratio (i.e., the first cost-performance ratio) under unknown workloads or dynamically changing workloads, and when , the formula degenerates to .

[0171] Phase 4: Configuration Selection. Select the UCPR maximum from all configurations:

[0172] (7);

[0173] wherein, represents the optimal disk combination (i.e., the second target disk configuration) under unknown workloads or dynamically changing workloads, which has good average performance and cost-effectiveness under various workloads.

[0174] Phase 5: Deployment. Deploy the optimal robust configuration (i.e., the second target disk configuration) to keep the system at a high cost-performance ratio under changing or unknown workloads.

[0175] Compared with the prior art, the embodiment has the following advantages and practical values:

[0176] First, in terms of performance improvement, the embodiment completely solves the resource competition problem between compression operations and write paths through physical separation of I / O paths. Experimental verification shows that 2.2 times and 2.1 times write throughput improvement is achieved on SSD and HDD respectively, far exceeding the local optimization effect of existing solutions in specific scenarios. More importantly, the embodiment establishes a complete cost-performance quantification model, which can systematically solve the storage configuration optimization problem in cloud environments, which is a key capability that existing technologies lack.

[0177] Second, in terms of practicality and scope of application, the embodiment provides a rigorous and easy-to-implement solution. Through the mathematical model based on the harmonic mean principle and the integral form of the general cost-performance metric, the embodiment not only supports accurate optimization of known workloads, but also handles unknown or changing workload scenarios that existing technologies cannot solve. Experimental results show that the disk configuration of ATLAS-CPR improves the average cost-performance ratio by 7.7% compared with the cheapest disk configuration under all test workloads, and the disk configuration of ATLAS-UCPR improves by an average of 5.7%, with almost no performance loss compared with the most expensive SSD and SSD disk configuration. This ability to significantly optimize cost-effectiveness while ensuring performance is a comprehensive advantage that existing storage optimization technologies cannot achieve.

[0178] The test and application of the technical solution of the embodiment include the following contents:

[0179] 1. Test environment configuration. The experimental verification of the embodiment is performed on Huawei cloud infrastructure, and a 4-node cluster configuration is adopted, including 3 TiKV data nodes and 1 benchmark test node. Each node is configured with 16 CPU cores, 32 GB of memory and 150 GB of local disk (single disk configuration of 300 GB), and runs CentOS 7.4.1708 (64-bit) operating system. The test uses the go-YCSB benchmark test suite, and configures 3 replica Raft consistency protocol and 100 concurrent client threads. Various YCSB workload modes are tested in the experiment, including Load (only insertion), Workload A (50% read / 50% update), Workload B (95% read / 5% update) and Workload C (only read), each record containing 16 bytes of key and 1 KB of value, and the performance is evaluated on a 100 GB complete data set.

[0180] 2. Typical application scenarios. The embodiment is suitable for various application scenarios sensitive to write performance and cost. Financial transaction systems require high concurrent write capability and strict delay SLO, and are sensitive to infrastructure cost, and the solution can provide stable write performance and optimize total cost of ownership. The Internet of Things data collection platform faces the real-time write demand of a large number of sensor data, and presents a typical read-less and write-more mode, and by optimizing the storage configuration, the operating cost can be significantly reduced. The log analysis platform needs to process real-time write of massive logs and periodic data analysis read, and the dual-disk architecture can effectively balance write performance and query efficiency. The e-commerce recommendation system requires real-time update of user behavior data and fast query of personalized recommendation, and the embodiment solution can control storage cost while ensuring real-time performance.

[0181] 3. Cloud service pricing model. The storage pricing model of major cloud service providers provides an important reference basis for the cost optimization of the present embodiment. Taking Huawei Cloud as an example, the pricing of the ultra-fast SSD cloud hard disk is 2.00 yuan / GiB / month, and the throughput is as high as 1000 MiB / s, while the general-purpose SSD V2 cloud hard disk is priced at only 0.50 yuan / GiB / month at the same throughput, and the high-IO HDD type is as low as 0.35 yuan / GiB / month. As for Tencent Cloud, the ultra-fast SSD and enhanced SSD are both 1.00 yuan / GiB / month, with a throughput of 4000 MB / s and 1000 MB / s respectively, and the general-purpose SSD is 0.50 yuan / GiB / month. Alibaba Cloud provides multi-level storage options from ESSD PL3 (4.00 yuan / GiB / month, 4000 MB / s) to efficient cloud disk (0.35 yuan / GiB / month, 140 MB / s). Such significant price and performance differences provide a rich configuration selection space for the cost-performance optimization model of the present embodiment, enabling the system to select the optimal storage configuration combination according to the specific workload requirements and budget constraints.

[0182] Reference Figure 4 The present embodiment also provides a distributed storage optimization system based on heterogeneous disks, which comprises a heterogeneous disk allocation unit 401, a performance benchmarking unit 402, a total cost calculation unit 403, a first cost-performance ratio calculation unit 404, a second cost-performance ratio calculation unit 405, a disk configuration determination unit 406, and a distributed storage unit 407, wherein:

[0183] The heterogeneous disk allocation unit 401 is configured to allocate a first disk for Raft log persistence and pre-write log write operations, and allocate a second disk for background compression read-write operations.

[0184] The performance benchmarking unit 402 is configured to perform performance benchmarking on all types of disks according to the first disk and the second disk, to obtain the write throughput and read throughput of each disk.

[0185] The total cost calculation unit 403 is configured to calculate the total cost of each disk combination according to the disk capacity and unit capacity price of each type of disk.

[0186] The first cost-performance ratio calculation unit 404 is configured to calculate the first cost-performance ratio of each disk combination under known workload conditions based on the total cost, write throughput, and read throughput.

[0187] The second cost-performance ratio calculation unit 405 is configured to calculate the second cost-performance ratio of each disk combination under unknown workload or changing workload conditions based on the total cost, write throughput, and read throughput.

[0188] The disk configuration determination unit 406 is configured to determine a first target disk configuration according to the respective first cost performance of all disk combinations, and determine a second target disk configuration according to the respective second cost performance of all disk combinations.

[0189] The distributed storage unit 407 is configured to perform distributed storage of data according to the first target disk configuration or the second target disk configuration.

[0190] It should be noted that, since the one kind of distributed storage optimization system based on heterogeneous disks in the embodiment and the one kind of distributed storage optimization method based on heterogeneous disks described above are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment, and details are not described herein.

[0191] Referring to Figure 5 The embodiment of the present application further provides an electronic device, and the electronic device comprises:

[0192] at least one memory;

[0193] at least one processor;

[0194] at least one program;

[0195] The program is stored in the memory, and the processor executes the at least one program to implement the distributed storage optimization method based on heterogeneous disks provided in the embodiment of the present disclosure.

[0196] The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0197] The electronic device of the embodiment of the present application is described in detail below.

[0198] The processor 1600 can be implemented in a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is configured to execute a related program to implement the technical solutions provided in the embodiment of the present disclosure.

[0199] The memory 1700 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 1700 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the present disclosure are implemented by software or firmware, the related program codes are stored in the memory 1700 and are invoked and executed by the processor 1600 to implement the heterogeneous disk-based distributed storage optimization method provided in the embodiments of the present disclosure.

[0200] The input / output interface 1800 is configured to realize information input and output.

[0201] The communication interface 1900 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0202] The bus 2000 is configured to transmit information between various components (for example, the processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900) of the device.

[0203] The processor 1600, the memory 1700, the input / output interface 1800, and the communication interface 1900 are connected to each other through the bus 2000 to realize the communication connection between the device.

[0204] The present disclosure further provides a storage medium, which is a computer readable storage medium, and stores computer executable instructions for causing a computer to execute the above-mentioned heterogeneous disk-based distributed storage optimization method.

[0205] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0206] The embodiments described in the present disclosure are to more clearly illustrate the technical solutions of the present disclosure, and do not constitute a limitation on the technical solutions provided by the present disclosure. Those skilled in the art can know that, as technology evolves and new application scenarios appear, the technical solutions provided by the present disclosure are also applicable to similar technical problems.

[0207] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the present disclosure, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0208] The device embodiments described above are merely illustrative, and units described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0209] Those skilled in the art can understand that all or some steps in the above disclosed method, the functions of the modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0210] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0211] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b and c can be single or multiple.

[0212] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0213] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0214] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0215] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various program storage media. The above combines the drawings to make a detailed description of the embodiments of the present application, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge range of those skilled in the art without departing from the purpose of the present application.

Claims

1. A heterogeneous disk based distributed storage optimization method, characterized in that, The method comprises: allocating a first disk for Raft log persistence and prewrite log write operation, and a second disk for background compression read-write operation, comprising: obtaining a write request sent by a client, and encapsulating the write request as a Raft log entry; copying the Raft log entry to a follower node; persisting the Raft log to the first disk; after a preset number of follower nodes are confirmed, the leader performs an apply operation to write data to a prewrite log; persisting the prewrite log to the first disk; performing performance benchmarking on all types of disks according to the first disk and the second disk to obtain the write throughput and read throughput of each disk; calculating the overall cost of each disk combination according to the disk capacity and unit capacity price of each type of disk; based on the overall cost, the write throughput and the read throughput, calculating a first cost performance of each disk combination under a known workload, and calculating a second cost performance of each disk combination under an unknown workload or a changing workload, comprising: ； ； in, Indicates the best value for money. This indicates the percentage of read operations in the workload. This indicates the percentage of write operations in the workload. Indicates the second cost-effectiveness. Indicates configuration The overall cost, Indicates the first The first disk is a disk of the class. Indicates the first Use a disk-like disk as a second disk. Indicates the first Disk-like read throughput Indicates the first Disk-like write throughput Represents the natural logarithm; determining a first target disk configuration according to the first cost performance of each disk combination, and determining a second target disk configuration according to the second cost performance of each disk combination; performing data distributed storage according to the first target disk configuration or the second target disk configuration.

2. The heterogeneous disk based distributed storage optimization method of claim 1, wherein, The performance benchmarking on all types of disks according to the first disk and the second disk to obtain the write throughput and read throughput of each disk, comprising: performing performance benchmarking on all types of disks as the first disk to obtain the write throughput of each disk; performing performance benchmarking on all types of disks as the second disk to obtain the read throughput of each disk.

3. The heterogeneous disk based distributed storage optimization method of claim 1, wherein, The calculation of the overall cost of each disk combination according to the disk capacity and unit capacity price of each type of disk, comprising: obtaining a server instance fixed cost; multiplying the disk capacity and unit capacity price of each type of disk to obtain the disk cost corresponding to each type of disk; adding the disk cost corresponding to each type of disk in each disk combination to obtain a cost addition result corresponding to each disk combination; adding the cost addition result corresponding to each disk combination and the server instance fixed cost to obtain the overall cost of each disk combination.

4. The heterogeneous disk based distributed storage optimization method of claim 1, wherein, The determination of a first target disk configuration according to the first cost performance of each disk combination, and the determination of a second target disk configuration according to the second cost performance of each disk combination, comprising: selecting a disk combination corresponding to the maximum first cost performance from the first cost performance of each disk combination as the first target disk configuration; selecting a disk combination corresponding to the maximum second cost performance from the second cost performance of each disk combination as the second target disk configuration.

5. A heterogeneous disk based distributed storage optimization system, comprising: A system for performing the heterogeneous disk-based distributed storage optimization method according to any one of claims 1 to 4, comprising: a heterogeneous disk allocation unit configured to allocate a first disk for Raft log persistence and pre-write log write operations, and a second disk for background compression read-write operations; a performance benchmarking unit configured to perform performance benchmarking on all types of disks according to the first disk and the second disk to obtain a write throughput and a read throughput of each disk; a total cost calculation unit configured to calculate a total cost of each disk combination according to a disk capacity and a unit capacity price of each type of disk; a first cost-performance calculation unit configured to calculate a first cost-performance of each disk combination under a known workload based on the total cost, the write throughput, and the read throughput; a second cost-performance calculation unit configured to calculate a second cost-performance of each disk combination under an unknown workload or a changing workload based on the total cost, the write throughput, and the read throughput; a disk configuration determination unit configured to determine a first target disk configuration according to the first cost-performance of each disk combination, and determine a second target disk configuration according to the second cost-performance of each disk combination; a distributed storage unit configured to perform data distributed storage according to the first target disk configuration or the second target disk configuration.

6. An electronic device, comprising: The computer readable storage medium stores computer executable instructions for causing a computer to perform the heterogeneous disk based distributed storage optimization method according to any one of claims 1 to 4.

7. A computer readable storage medium characterized in that, The computer readable storage medium stores computer executable instructions for causing a computer to perform the heterogeneous disk based distributed storage optimization method according to any one of claims 1 to 4.

Images