High cost performance dynamic hierarchical storage method and system based on STaaS cloud

Abstract

The application provides a high cost performance dynamic hierarchical storage method and system based on STaaS cloud, acquires request load data, STaaS cloud price data and storage category data of a current time slot; obtains a cost generated in the current time slot according to the request load data and the STaaS cloud price data of the current time slot; obtains an unsatisfied delay according to delay levels provided by different storage category data and request delay constraints; calculates a storage category with the highest cost performance based on a weighted sum of the unsatisfied delay and the cost; judges whether the storage category with the highest cost performance is a current storage category, if yes, data is not migrated, and if not, the data is migrated to the storage category with the highest cost performance; the application can dynamically adjust the storage category of the data according to the requests reached in an online manner without knowing the access mode of the data in the whole life cycle, and can obtain an approximately optimal cost while meeting the delay demand of the requests.

Description

Technical Field

[0001] This invention relates to the field of cloud storage technology, and in particular to a cost-effective dynamic tiered storage method and system based on STaaS cloud. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, public cloud service providers have offered STaaS (Storage as a Service) storage service models, exemplified by object storage services. STaaS clouds provide users with an out-of-the-box approach to using storage services. Cloud storage users can host their data in the cloud without having to purchase and maintain their own data centers, saving significant manpower and material costs. While hosting data in the cloud improves maintenance convenience and cost-effectiveness, the cost of using STaaS clouds remains substantial as data volumes increase. Therefore, reducing the cost of using STaaS clouds has become a primary concern for STaaS users.

[0004] Currently, cloud service providers offer various storage categories for their storage services, each with different prices and access performance. For example, a certain storage service offers Standard, Standard-Infrequent Access, and Glacier storage. From Standard to Glacier, storage prices gradually increase, but access costs gradually decrease. Based on the pricing differences between storage categories, appropriate storage tiers can be selected for data with different access patterns to reduce costs. Beyond pricing differences, different storage categories also offer varying latency: Standard and Standard-Infrequent Access offer millisecond-level request latency, while Glacier offers minute-level latency. Therefore, while utilizing multiple storage categories to optimize usage costs, it's also necessary to consider the impact of different storage categories on access performance. Blindly storing data in the lowest-cost storage category may reduce usage costs, but it could increase user request latency, thereby degrading the user experience and resulting in more harm than good.

[0005] For frequently accessed and relatively infrequently accessed data, standard storage and low-frequency storage can be used respectively to reduce usage costs while ensuring low request latency. For infrequently accessed data, archive storage offers the lowest usage cost but increases request latency, impacting user experience. For this type of data, the appropriate storage category can be selected based on request latency requirements: requests with high latency requirements should be stored in low-frequency storage to ensure low latency; while requests with less stringent latency requirements should be stored in archive storage to reduce usage costs.

[0006] Therefore, to achieve optimal cost while ensuring request performance, it's crucial to accurately understand access patterns throughout the data's lifecycle. However, user access to data occurs online, and access volume gradually emerges over time. Future access volume cannot be predicted in advance, and it's difficult to accurately forecast. In this situation, users can only adjust storage categories based on historical access patterns. However, adjusting data categories based on historical access patterns rarely achieves optimal cost while maintaining request latency. For example, suppose data was frequently accessed in the past. To achieve optimal cost, the data might be stored in the standard storage category for a future period. Clearly, if the data continues to be frequently accessed in the future, this decision can reduce costs. However, if the data is not frequently accessed in the future, this decision will not only fail to reduce costs but may even incur higher costs.

[0007] Existing technologies typically employ predictive methods to adjust storage categories. However, since data access patterns and latency requirements cannot be accurately predicted, the storage category adjustment decisions made cannot always guarantee cost savings and performance guarantees. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a cost-effective dynamic tiered storage method and system based on STaaS cloud. It does not require prior knowledge of the access patterns throughout the entire data lifecycle and can dynamically adjust the storage category of data based on requests arriving online, achieving near-optimal cost while meeting the latency requirements of requests.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] The first aspect of this invention provides a cost-effective dynamic tiered storage method based on STaaS cloud.

[0011] A cost-effective dynamic tiered storage method based on STaaS cloud includes the following processes:

[0012] Obtain request load data, STaaS cloud pricing data, and storage category data for the current time slot;

[0013] Based on the current time slot request load data and STaaS cloud pricing data, the cost generated in the current time slot is obtained;

[0014] Based on the latency levels and request latency constraints provided by different storage categories, the unmet latency is obtained;

[0015] Based on the weighted sum of unmet latency and cost, the storage category with the highest cost-performance ratio is calculated.

[0016] Determine if the storage category with the best cost-performance ratio is the current storage category. If so, the data will not be migrated; otherwise, the data will be migrated to the storage category with the best cost-performance ratio.

[0017] As a further limitation of the first aspect of the present invention, the number of requests occurring in the current time slot and the total latency are obtained, and the average latency of the requests is calculated;

[0018] The cumulative unmet update latency is obtained based on the calculated average latency of each data object receiving the request;

[0019] The calculation will update the accumulated unmet latency and use it for the calculation of the most cost-effective storage class in the next time slot.

[0020] As a further limitation of the first aspect of the present invention, the storage class with the highest cost performance is calculated, including: e = argmin{q(lL) + VC}, where q represents the cumulative unmet latency, l represents the average latency of the current time slot, L represents the latency upper limit constraint, V is a weight parameter, and C represents the cost generated by the current time slot.

[0021] As a further limitation of the first aspect of the invention, the cumulative unmet delay is updated based on the calculated average delay at which each data object receives a request, including:

[0022] q' = max{q + lL, 0}

[0023] Where l represents the average delay of the current time slot, L represents the upper limit constraint of delay, q' is the cumulative unmet delay of the current time slot update, and q is the cumulative unmet delay of the previous time slot update.

[0024] As a further limitation of the first aspect of the present invention, the request load data includes: the number of requests in the current time slot, the amount of data retrieved, and the data size.

[0025] As a further limitation of the first aspect of the present invention, STaaS cloud pricing data includes: storage price, request price, data retrieval price, and data migration price.

[0026] The second aspect of this invention provides a cost-effective dynamic tiered storage system based on STaaS cloud.

[0027] A cost-effective dynamic tiered storage system based on STaaS cloud includes:

[0028] The data acquisition unit is configured as follows:

[0029] Obtain request load data, STaaS cloud pricing data, and storage category data for the current time slot;

[0030] The cost optimization unit is configured as follows:

[0031] Based on the current time slot request load data and STaaS cloud pricing data, the cost generated in the current time slot is obtained;

[0032] Based on the latency levels and request latency constraints provided by different storage categories, the unmet latency is obtained;

[0033] Based on the weighted sum of unmet latency and cost, the storage category with the highest cost-performance ratio is calculated.

[0034] Determine if the storage category with the best cost-performance ratio is the current storage category. If so, the data will not be migrated; otherwise, the data will be migrated to the storage category with the best cost-performance ratio.

[0035] As a further limitation of the second aspect of the present invention, the number of requests occurring in the current time slot and the total latency are obtained, and the average latency of the requests is calculated;

[0036] The cumulative unmet update latency is obtained based on the calculated average latency of each data object receiving the request;

[0037] The calculation will update the accumulated unmet latency and use it for the calculation of the most cost-effective storage class in the next time slot.

[0038] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the cost-effective dynamic tiered storage method based on STaaS cloud as described in the first aspect of the present invention.

[0039] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the cost-effective dynamic tiered storage method based on STaaS cloud as described in the first aspect of the present invention.

[0040] Compared with the prior art, the beneficial effects of the present invention are:

[0041] 1. This invention innovatively proposes a cost-effective dynamic tiered storage method and system based on STaaS cloud. It does not require prior knowledge of the access patterns throughout the entire data lifecycle. It can dynamically adjust the storage category of data based on requests arriving online, and can achieve near-optimal cost while meeting the latency requirements of requests.

[0042] 2. This invention innovatively proposes a cost-effective dynamic tiered storage method and system based on STaaS cloud. It constructs a tiered storage strategy with performance and price differences based on STaaS cloud, and can make storage category adjustment decisions based only on short-term data access volume.

[0043] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1 This is a flowchart illustrating the cost-effective dynamic tiered storage method based on STaaS cloud provided in Embodiment 1 of the present invention.

[0046] Figure 2 This is a schematic diagram of the architecture of a cost-effective dynamic tiered storage system based on STaaS cloud provided in Embodiment 2 of the present invention. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0050] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0051] Example 1:

[0052] like Figure 1 As shown, Embodiment 1 of the present invention provides a cost-effective tiered storage method based on STaaS cloud. Taking Amazon S3 service as an example, for each data object stored therein, the following process is included:

[0053] Step 1: Initialize the cumulative unmet delay q = 0.

[0054] Step 2: Collect request load, price, and storage category information in real time.

[0055] Step 2.1: Price information collection.

[0056] A web crawler is used to collect S3 prices in real time, with the collection interval in time slots. In this embodiment, one time slot represents one day. The web crawler is executed at a set time every morning to download, parse, and save the storage price, request price, data retrieval price, and data migration price from the S3 service interface into a MySQL database.

[0057] It should be noted that the crawler program used in this embodiment is a legitimate data collection program, and the collected data is all legally public and can be legally used.

[0058] Step 2.2: Request load information collection.

[0059] The request load on data objects is collected in real time, with collection intervals measured in days. Taking Amazon S3 as an example, load collection specifically includes the following steps: Configure a logging function for the data object using Amazon S3 Object Lambda, add the logging code to the S3 GET request, configure an access path for the Lambda function, expose this path to users, and allow users to request the data object through this path; after receiving the request, the Lambda function asynchronously sends the request time, the amount of data retrieved, and other data to a log server deployed on EC2 in the same region, and the log server stores the data in a MySQL database.

[0060] Step 2.3: Collect the storage category of each data object in real time.

[0061] The storage category of a data object is stored in its metadata. After each request, the storage category of the object is parsed from the metadata, and the latest storage category is stored in the MySQL database.

[0062] Step 3: Calculate the latest storage class.

[0063] Step 3.1: Calculate the cost C generated in the current time slot based on the latest collected price;

[0064] Step 3.2: Calculate the unmet latency lL based on the latency level l and request latency constraint L provided by different storage types;

[0065] Step 3.3: Based on the weighted sum of unmet latency and cost requirements, calculate the storage class with the highest cost-performance ratio. The calculation formula is as follows:

[0066] e = argmin{q(lL) + VC}(1)

[0067] Where q represents the cumulative unmet delay, l represents the average delay of the current time slot, L represents the delay upper limit constraint, V is the weighting parameter, and C represents the cost incurred by the current time slot, including storage cost C. s Request cost C g Search cost C r And data migration cost C m :

[0068] C = C s +C g +C r +C m (2)

[0069] Among them, storage cost C s Calculated as follows:

[0070]

[0071] Among them, S h S c S g These represent the unit prices for standard storage, infrequent storage, and archive storage, respectively, with S representing the data size.

[0072] Request cost C g Calculated in the following way:

[0073]

[0074] G h G c G g These represent the unit price for standard storage, infrequent storage, and archive storage, respectively, with G representing the number of requests.

[0075] Search cost C r Calculated in the following way:

[0076]

[0077] R h R c R gThese represent the retrieval unit price for standard storage, low-frequency storage, and archive storage, respectively, and R represents the amount of data retrieved.

[0078] Data migration cost C m Calculated in the following way:

[0079]

[0080] Step 4: Based on the storage category with the highest cost-performance ratio calculated, migrate the data to the new storage category using the SDK.

[0081] Step 5: Request performance monitoring and average latency calculation.

[0082] Enable the S3 server access log function. The log records the turnaround time for each object in each bucket to receive each request, thereby representing the latency of requesting to retrieve data objects. The access log is stored in the S3 bucket. By analyzing the access log, the number of requests occurring in the current time slot and the total latency are counted, and the average latency of the request is calculated.

[0083] Step 6: Calculate and update the cumulative unmet latency based on the calculated average latency of receiving requests for each data object:

[0084] q=max{q+lL,0}(7)

[0085] Where l represents the average delay of the current time slot, and L represents the upper limit constraint on delay.

[0086] Step 7: Proceed to the next optimization slot and repeat steps 2-6.

[0087] Example 2:

[0088] like Figure 2 As shown, Embodiment 2 of the present invention provides a high-performance, cost-effective dynamic tiered storage system based on STaaS cloud, comprising:

[0089] The data acquisition unit is configured as follows:

[0090] Obtain request load data, STaaS cloud pricing data, and storage category data for the current time slot;

[0091] The cost optimization unit is configured as follows:

[0092] Based on the current time slot request load data and STaaS cloud pricing data, the cost generated in the current time slot is obtained;

[0093] Based on the latency levels and request latency constraints provided by different storage categories, the unmet latency is obtained;

[0094] Based on the weighted sum of unmet latency and cost, the storage category with the highest cost-performance ratio is calculated.

[0095] Determine if the storage category with the best cost-performance ratio is the current storage category. If so, the data will not be migrated. If not, the data will be migrated to the storage category with the best cost-performance ratio via the SDK.

[0096] The data acquisition unit mainly includes: a price information acquisition module, a request information acquisition module, a performance monitoring module, and a storage category monitoring module.

[0097] The price information collection module includes: using a web crawler to download and parse the prices of STaaS cloud services, and saving the parsed storage prices, request prices, data retrieval prices and data migration prices to a database. The web crawler is executed periodically to ensure that price changes can be captured in a timely manner.

[0098] The request information collection module includes: real-time recording of access logs for each data object, with each log including access time and the amount of data retrieved, and saving the data. This invention is developed and deployed in a user-insensitive manner, that is, it completes the collection and recording of request information based on the relevant services provided by the STaaS cloud, without needing to deploy components or middleware on the user end.

[0099] For example, when using Amazon S3 services, this module can be implemented based on S3 Object Lambda. By taking advantage of the feature that Lambda functions are automatically executed along with requests, the information of the requests being processed is recorded before the S3 service responds to each request, and the information is saved to the database server. This method requires renting an additional EC2 database server.

[0100] In addition, the access log function provided by STaaS cloud can be used. For example, when applying Alibaba Cloud OSS service, this module can be implemented based on access logs. After enabling the OSS log management function, all requests to the data stored there will be recorded. Using the SDK, the generated access logs are periodically transferred to a specific storage bucket of the OSS service. Unlike the Lambda-based implementation, this method does not require purchasing additional EC2 instances to deploy a database server, but transferring access logs to the OSS service incurs additional storage costs. Since the method provided by this invention only requires request information from the most recent time slot, older request information can be deleted to save log storage costs.

[0101] The storage category monitoring module includes: checking the storage category of each object within each optimization slot, which is achieved by viewing the metadata of the data through the SDK.

[0102] The performance monitoring module includes: collecting the latency of each request, where latency refers to the time difference between when the STaaS cloud service receives a request and when it responds to the request. This module can be implemented based on the access logs provided by the STaaS cloud service: obtaining the log file stored in the cloud service through the SDK, parsing the log file, and extracting the time consumed to complete the request.

[0103] In the cost optimization unit, the most cost-effective storage category is calculated based on price, request information, request latency, and storage category. Data objects are then migrated to the new storage category. Information collected from the database or STaaS cloud is obtained to calculate the most cost-effective storage category for the current time slot. The storage category of the data is then adjusted through the SDK.

[0104] The implementation of each of the above units is based on the characteristics of STaaS cloud. They can run independently outside of STaaS cloud services and STaaS cloud users, do not depend on user-side configuration, do not require STaaS cloud users to modify existing applications, and will not affect existing business.

[0105] Example 3:

[0106] Embodiment 3 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the cost-effective dynamic tiered storage method based on STaaS cloud as described in Embodiment 1 of the present invention.

[0107] Example 4:

[0108] Embodiment 4 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the cost-effective dynamic tiered storage method based on STaaS cloud as described in Embodiment 1 of the present invention.

[0109] In this embodiment, taking Amazon Web Services (AWS), a leading cloud service provider, as an example, and combining the various features it provides in terms of storage services, the implementation steps of the technical solution proposed in this invention are described in detail. This solution enables dynamic adjustment of data storage categories based on request load to optimize costs as much as possible while ensuring latency constraints, and has strong representativeness and wide applicability.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cost-effective dynamic tiered storage method based on STaaS cloud, characterized in that, Includes the following processes: Obtain the request load data, STaaS cloud pricing data, and storage category data for the current time slot; based on the request load data and STaaS cloud pricing data for the current time slot, calculate the cost incurred in the current time slot. Based on the latency levels and request latency constraints provided by different storage categories, the unmet latency is obtained; Based on the weighted sum of unmet latency and cost, the storage category with the highest cost-performance ratio is calculated. The storage classes that offer the best value for money include: ,in, q This indicates that the cumulative delay has not been met. l This represents the average delay of the current time slot. L This indicates a delay upper limit constraint. V For weight parameters, C This represents the cost incurred in the current time slot; Get the number of requests and total latency that occurred in the current time slot, and calculate the average latency of the requests; Based on the calculated average latency for each data object to receive a request, the cumulative unmet latency for updating is obtained, including: in, l This represents the average delay of the current time slot. L This indicates a delay upper limit constraint. q’ The accumulated unmet delay for the update of the current time slot, q The accumulated delay for the update in the previous time slot; The updated cumulative unmet latency will be used to calculate the most cost-effective storage class for the next time slot; Determine if the storage category with the best cost-performance ratio is the current storage category. If so, the data will not be migrated; otherwise, the data will be migrated to the storage category with the best cost-performance ratio.

2. The cost-effective dynamic tiered storage method based on STaaS cloud as described in claim 1, characterized in that, Request load data, including: the number of requests in the current time slot, the amount of data retrieved, and the data size.

3. The cost-effective dynamic tiered storage method based on STaaS cloud as described in claim 1, characterized in that, STaaS cloud pricing data includes: storage price, request price, data retrieval price, and data migration price.

4. A cost-effective dynamic tiered storage system based on STaaS cloud, characterized in that, include: The data acquisition unit is configured as follows: Retrieve request load data, STaaS cloud pricing data, and storage category data for the current time slot; The cost optimization unit is configured as follows: Based on the current time slot request load data and STaaS cloud pricing data, the cost generated in the current time slot is obtained; Based on the latency levels and request latency constraints provided by different storage categories, the unmet latency is obtained; Based on the weighted sum of unmet latency and cost, the storage category with the highest cost-performance ratio is calculated. The storage classes that offer the best value for money include: ,in, q This indicates that the cumulative delay has not been met. l This represents the average delay of the current time slot. L This indicates a delay upper limit constraint. V For weight parameters, C This represents the cost incurred in the current time slot; Get the number of requests and total latency that occurred in the current time slot, and calculate the average latency of the requests; Based on the calculated average latency for each data object to receive a request, the cumulative unmet latency for updating is obtained, including: in, l This represents the average delay of the current time slot. L This indicates a delay upper limit constraint. q’ The accumulated unmet delay for the update of the current time slot, q The accumulated delay for the update in the previous time slot; The updated cumulative unmet latency will be used to calculate the most cost-effective storage class for the next time slot; Determine if the storage category with the best cost-performance ratio is the current storage category. If so, the data will not be migrated; otherwise, the data will be migrated to the storage category with the best cost-performance ratio.

5. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps of the cost-effective dynamic tiered storage method based on STaaS cloud as described in any one of claims 1-3.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the cost-effective dynamic tiered storage method based on STaaS cloud as described in any one of claims 1-3.

Images