Cloud storage optimization system and method based on intelligent grading

Abstract

The invention relates to the technical field of cloud storage optimization, in particular to a cloud storage optimization system and method based on intelligent classification. The method comprises the steps that to-be-stored data information is obtained, and the to-be-stored data information comprises the access frequency of to-be-stored data, the timeliness of the to-be-stored data, the storage response delay of the to-be-stored data, the size of a storage space needed by the to-be-stored data and the availability of the to-be-stored data; generating a to-be-stored data classification index CCZ based on the to-be-stored data information; performing grading processing on the to-be-stored data based on the to-be-stored data grading index CCZ, and determining a storage grade corresponding to the to-be-stored data; and performing cloud storage on the to-be-stored data based on the level. According to the invention, the performance of the cloud storage system can be improved, the cost is reduced, and the reliability and availability of data are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud storage optimization, and in particular to a cloud storage optimization system and method based on intelligent grading. Background Art

[0002] With the rapid development of cloud computing technology, cloud storage has become a vital infrastructure for data storage and management for both businesses and individuals. With its advantages of high scalability, high availability, and data backup and recovery, cloud storage meets users' growing data storage needs. However, with the explosive growth of data volumes and the increasing complexity of data processing, traditional cloud storage methods are gradually facing performance bottlenecks and cost challenges.

[0003] For hot data with high access frequency, if it is still stored on storage media with low performance, it will seriously affect the response speed of data access and user experience; for data with low access frequency, if it is stored on high-performance storage media, it will cause waste of resources and increase costs.

[0004] Therefore, there is an urgent need for a cloud storage optimization system and method based on intelligent tiering to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a cloud storage optimization system and method based on intelligent grading: to solve the technical problem that existing solutions cannot meet the diverse requirements of storage performance, cost and reliability in different application scenarios in traditional cloud storage methods.

[0006] The purpose of the present invention can be achieved through the following technical solutions: On the one hand, a cloud storage optimization system based on intelligent tiering includes a module for acquiring information about data to be stored, a module for calculating tiering indexes, a module for tiering data to be stored, and a storage deployment module; The module for acquiring information about data to be stored is used to acquire information about data to be stored and send the information about data to be stored to the grading index calculation module, wherein the information about data to be stored includes the access frequency of the data to be stored, the timeliness of the data to be stored, the storage response delay of the data to be stored, the storage space required for the data to be stored, and the availability of the data to be stored; The grading index calculation module is used to receive the data information to be stored, generate the grading index CCZ of the data to be stored based on the data information to be stored, and send the grading index CCZ of the data to be stored to the grading module of the data to be stored; The data to be stored classification module is used to receive the classification index CCZ of the data to be stored, and classify the data to be stored based on the classification index CCZ of the data to be stored, determine the storage level corresponding to the data to be stored, and send the storage level corresponding to the data to be stored to the storage deployment module; The storage deployment module is used to receive the storage level corresponding to the data to be stored, and perform cloud storage on the data to be stored based on the level.

[0007] Furthermore, generating the to-be-stored data classification index CCZ based on the to-be-stored data information specifically includes the following process: Acquire an access frequency FW of the data to be stored based on the information of the data to be stored, wherein the access frequency of the data to be stored is the number of times the data to be stored is accessed per unit time; Obtaining a timeliness rate SX of the data to be stored based on the information of the data to be stored, wherein the timeliness rate of the data to be stored is obtained by using a search engine to obtain a timeliness parameter of each data to be stored, and using the timeliness parameter as the timeliness rate SX of the data to be stored; Obtaining a storage response delay YS of the data to be stored based on the information of the data to be stored, wherein the storage response delay of the data to be stored is a time interval between when the storage node issues a request to store the data and when it responds to the request to store the data; Obtain the storage space size KJ required for the data to be stored based on the data to be stored information; The availability KY of the data to be stored is calculated based on the information of the data to be stored; Substitute the access frequency FW, the update rate SX, the storage response delay YS, the required storage space size KJ, and the availability KY into the calculation formula of the data to be stored classification index to obtain the data to be stored classification index CCZ. The calculation formula is as follows: ; in, 、 、 are weight coefficients, which are 0.3, 0.4, and 0.3 respectively. The weight coefficients are used to measure the influence of different data to be stored on the classification of the data to be stored. The value of is 2.72.

[0008] Furthermore, calculating the availability KY of the data to be stored based on the information of the data to be stored specifically includes the following process: Based on the information of the data to be stored, the MTTF (Mean Time to Failure to Use) and the MTTR (Mean Time to Repair) of the data to be stored are obtained, wherein the MTTR (Mean Time to Repair) of the data to be stored is the time required to repair the data to be stored based on the erasure coding technology; Calculate the availability KY of the data to be stored: .

[0009] Furthermore, the data to be stored is graded based on the grade index CCZ of the data to be stored, and the storage level corresponding to the data to be stored is determined, specifically including the following process: Load the grading index threshold of the data to be stored and ,in, and is pre-stored in the system, and Greater than ; If the CCZ of the data to be stored is less than , then the storage level corresponding to the data to be stored is the first-level storage level; If the CCZ of the data to be stored is greater than and less than , then the storage level corresponding to the data to be stored is the secondary storage level; If the CCZ of the data to be stored is greater than , then the storage level corresponding to the data to be stored is the third-level storage level.

[0010] Furthermore, the cloud storage of the stored data based on the level specifically includes the following processes: Step 1: Count the cache space of each node used to cache the data to be stored and gain , according to the unit space gain of each node Build a node priority queue in ascending order; Step 2: Get the probability that the total number of cache requests for data to be stored within time T is k and the data content size of the data to be stored , where the cache request for data to be stored within time T is recorded as a Poisson process; Step 3: Based on probability Build a content priority queue for data to be stored; Step 4: Determine whether the content priority queue of the data to be stored is empty. If so, the cloud storage ends. If not, the data to be stored at the top of the content priority queue of the data to be stored is used as the current cache object, and traverse in the order of the node priority queue to match the data content size of the current cache object. If the node exists, the current cache object is deployed on the node; if the node does not exist, the current cache object is removed from the content priority queue of the data to be stored and stored in the specified node, wherein the specified node is not in the node priority queue; Step 5: Repeat steps 2 to 4 until the content priority queue of the data to be stored is empty.

[0011] Furthermore, the probability that the total number of cache requests for data to be stored within time T is k is obtained The specific process includes the following: ; in, It is the average number of requests to the cache for data to be stored per unit time.

[0012] Furthermore, the system also includes a data storage update module. The workflow of the data storage update module is as follows: Initiate an update request: The user identifies the data that needs to be updated and initiates a data update request to the cloud server through the cloud storage service's API. The request includes the identifier of the data to be updated, the new data content or incremental data, and necessary authentication information. The cloud storage server updates the data in the storage system according to the user's request. The update process includes overwriting the original data, appending data to the end of the file, or updating the records in the database.

[0013] In another aspect, a cloud storage optimization method based on intelligent tiering includes: Acquiring information about the data to be stored, wherein the information about the data to be stored includes an access frequency of the data to be stored, a freshness rate of the data to be stored, a storage response delay of the data to be stored, a storage space required for the data to be stored, and availability of the data to be stored; Generate a classification index CCZ of data to be stored based on information of data to be stored; The data to be stored is graded based on the classification index CCZ of the data to be stored, and the storage level corresponding to the data to be stored is determined; Cloud storage based on the level of storage data.

[0014] Compared with the existing solutions, the present invention achieves the following beneficial effects: The present invention performs intelligent analysis on the data to be stored in the cloud storage system, and hierarchically stores the data to be stored based on the data information to be stored, including the access frequency of the data to be stored, the timeliness of the data to be stored, the storage response delay of the data to be stored, the storage space required for the data to be stored, and the availability of the data to be stored. This intelligent hierarchical storage strategy can improve the performance of the cloud storage system, reduce costs, and enhance the reliability and availability of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments described in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0016] Figure 1 This is a system block diagram of a cloud storage optimization system based on intelligent tiering according to an embodiment of the present invention; Figure 2 This is a workflow diagram of a cloud storage optimization system based on intelligent tiering according to an embodiment of the present invention; Figure 3 This is a workflow diagram of a cloud storage optimization method based on intelligent tiering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] In addition, the described features, structures or characteristics can be combined in any suitable manner in one or more example embodiments. In the following description, many specific details are provided to provide a full understanding of the example embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced while omitting one or more of the specific details, or other methods, components, steps, etc. can be adopted. In other cases, well-known structures, methods, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0019] This embodiment provides a cloud storage optimization system based on intelligent grading. Figure 1 This is a system block diagram of a cloud storage optimization system based on intelligent grading according to an embodiment of the present invention. Figure 1 As shown, the system includes a module for acquiring information about data to be stored, a module for calculating a grading index, a module for grading data to be stored, and a storage deployment module; The module for acquiring information about data to be stored is used to acquire information about data to be stored and send the information about data to be stored to the grading index calculation module, wherein the information about data to be stored includes the access frequency of the data to be stored, the timeliness of the data to be stored, the storage response delay of the data to be stored, the storage space required for the data to be stored, and the availability of the data to be stored; The grading index calculation module is used to receive the data information to be stored, generate the grading index CCZ of the data to be stored based on the data information to be stored, and send the grading index CCZ of the data to be stored to the grading module of the data to be stored; The data to be stored classification module is used to receive the classification index CCZ of the data to be stored, and classify the data to be stored based on the classification index CCZ of the data to be stored, determine the storage level corresponding to the data to be stored, and send the storage level corresponding to the data to be stored to the storage deployment module; The storage deployment module is used to receive the storage level corresponding to the data to be stored, and perform cloud storage on the data to be stored based on the level.

[0020] In summary, the present invention performs intelligent analysis on the data to be stored in the cloud storage system, and performs hierarchical storage on the data to be stored based on the data information to be stored, including the access frequency of the data to be stored, the timeliness of the data to be stored, the storage response delay of the data to be stored, the size of the storage space required for the data to be stored, and the availability of the data to be stored. This intelligent hierarchical storage strategy can improve the performance of the cloud storage system, reduce costs, and enhance the reliability and availability of data.

[0021] In some embodiments, Figure 2 This is a workflow diagram of a cloud storage optimization system based on intelligent grading according to an embodiment of the present invention. Figure 2 As shown, generating the to-be-stored data classification index CCZ based on the to-be-stored data information specifically includes the following steps: Step S201: obtaining an access frequency FW of the data to be stored based on the information of the data to be stored, wherein the access frequency of the data to be stored is the number of times the data to be stored is accessed per unit time; Step S202: obtaining the update rate SX of the data to be stored based on the information of the data to be stored, wherein the update rate of the data to be stored is obtained by using a search engine to obtain the update parameter of each data to be stored, and using the update parameter as the update rate SX of the data to be stored; It is worth noting that the timeliness parameter of each data to be stored is a feature of the data, which describes the freshness of the data or the timestamp of the most recent access, update, or generation. The timeliness parameter value can be obtained through a search engine and is stored in the system. Step S203: obtaining a storage response delay YS of the data to be stored based on the information of the data to be stored, wherein the storage response delay of the data to be stored is the time interval between when the storage node issues a request to store the data and when it responds to the request to store the data; Step S204: obtaining the storage space size KJ required for the data to be stored based on the information of the data to be stored; Step S205: Calculate the availability KY of the data to be stored based on the information of the data to be stored; Step S206: Substitute the access frequency FW, the update rate SX, the storage response delay YS, the required storage space size KJ, and the availability KY into the calculation formula of the data to be stored classification index to obtain the data to be stored classification index CCZ. The calculation formula is as follows: ; in, 、 、 are weight coefficients, which are 0.3, 0.4, and 0.3 respectively. The weight coefficients are used to measure the influence of different data to be stored on the classification of the data to be stored. The value of is 2.72.

[0022] In some embodiments, in step S205, calculating the availability KY of the data to be stored based on the information of the data to be stored specifically includes the following process: Based on the information of the data to be stored, the MTTF (Mean Time to Failure to Use) and the MTTR (Mean Time to Repair) of the data to be stored are obtained, wherein the MTTR (Mean Time to Repair) of the data to be stored is the time required to repair the data to be stored based on the erasure coding technology; Calculate the availability KY of the data to be stored: .

[0023] It is worth noting that erasure coding technology is a forward error correction technology (FEC), which plays an important role in data protection, improving storage reliability, and avoiding packet loss in network transmission.

[0024] In some embodiments, the data to be stored is graded based on the to-be-stored data grade index CCZ, and the determination of the storage level corresponding to the data to be stored specifically includes the following process: Load the grading index threshold of the data to be stored and ,in, and is pre-stored in the system, and Greater than ; If the CCZ of the data to be stored is less than , then the storage level corresponding to the data to be stored is the first-level storage level; If the CCZ of the data to be stored is greater than and less than , then the storage level corresponding to the data to be stored is the secondary storage level; If the CCZ of the data to be stored is greater than , then the storage level corresponding to the data to be stored is the third-level storage level.

[0025] It is worth mentioning that each storage level corresponds to a different storage node.

[0026] In some embodiments, cloud storage of data to be stored based on level specifically includes the following process: Step 1: Count the cache space of each node used to cache the data to be stored and gain , according to the unit space gain of each node Build a node priority queue in ascending order; Step 2: Get the probability that the total number of cache requests for data to be stored within time T is k and the data content size of the data to be stored , where the cache request for data to be stored within time T is recorded as a Poisson process; Step 3: Based on probability Build a content priority queue for data to be stored; Step 4: Determine whether the content priority queue of the data to be stored is empty. If so, the cloud storage ends. If not, the data to be stored at the top of the content priority queue of the data to be stored is used as the current cache object, and traverse in the order of the node priority queue to match the data content size of the current cache object. If the node exists, the current cache object is deployed on the node; if the node does not exist, the current cache object is removed from the content priority queue of the data to be stored and stored in the specified node, wherein the specified node is not in the node priority queue; Step 5: Repeat steps 2 to 4 until the content priority queue of the data to be stored is empty.

[0027] Furthermore, the probability that the total number of cache requests for data to be stored within time T is k is obtained The specific process includes the following: ; in, It is the average number of requests to the cache for data to be stored per unit time.

[0028] In some embodiments, the system further includes a data storage update module. The specific working process of the data storage update module is as follows: Initiate an update request: The user identifies the data that needs to be updated and initiates a data update request to the cloud server through the cloud storage service's API. The request includes the identifier of the data to be updated, the new data content or incremental data, and necessary authentication information. The cloud storage server updates the data in the storage system according to the user's request. The update process includes overwriting the original data, appending data to the end of the file, or updating the records in the database.

[0029] This embodiment also provides a cloud storage optimization method based on intelligent grading. Figure 3 This is a workflow diagram of a cloud storage optimization method based on intelligent grading according to an embodiment of the present invention. Figure 3 As shown, the method includes: Step S301: Acquire information about data to be stored, wherein the information about the data to be stored includes access frequency of the data to be stored, update rate of the data to be stored, storage response delay of the data to be stored, storage space required for the data to be stored, and availability of the data to be stored; Step S302: generating a classification index CCZ of data to be stored based on the information of the data to be stored; Step S303: performing classification processing on the data to be stored based on the classification index CCZ of the data to be stored, and determining the storage level corresponding to the data to be stored; Step S304: Cloud storage is performed on the data to be stored based on the level.

[0030] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in the embodiments of this application are fully or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0031] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0032] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0033] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only for some logical functions. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0034] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0035] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A cloud storage optimization system based on intelligent grading, characterized in that: The system includes a module for acquiring information about data to be stored, a module for calculating a grading index, a module for grading data to be stored, and a storage deployment module; The module for acquiring information about data to be stored is used to acquire information about data to be stored and send the information about data to be stored to the grading index calculation module, wherein the information about data to be stored includes the access frequency of the data to be stored, the timeliness of the data to be stored, the storage response delay of the data to be stored, the storage space required for the data to be stored, and the availability of the data to be stored; The grading index calculation module is used to receive the data information to be stored, generate the grading index CCZ of the data to be stored based on the data information to be stored, and send the grading index CCZ of the data to be stored to the grading module of the data to be stored; The data to be stored classification module is used to receive the classification index CCZ of the data to be stored, and classify the data to be stored based on the classification index CCZ of the data to be stored, determine the storage level corresponding to the data to be stored, and send the storage level corresponding to the data to be stored to the storage deployment module; The storage deployment module is used to receive the storage level corresponding to the data to be stored, and perform cloud storage on the data to be stored based on the level.

2. The cloud storage optimization system based on intelligent grading according to claim 1, characterized in that: Generate the data classification index CCZ of the data to be stored based on the data to be stored information The following processes are included: Acquire an access frequency FW of the data to be stored based on the information of the data to be stored, wherein the access frequency of the data to be stored is the number of times the data to be stored is accessed per unit time; Obtaining a timeliness rate SX of the data to be stored based on the information of the data to be stored, wherein the timeliness rate of the data to be stored is obtained by using a search engine to obtain a timeliness parameter of each data to be stored, and using the timeliness parameter as the timeliness rate SX of the data to be stored; Obtaining a storage response delay YS of the data to be stored based on the information of the data to be stored, wherein the storage response delay of the data to be stored is a time interval between when the storage node issues a request to store the data and when it responds to the request to store the data; Obtain the storage space size KJ required for the data to be stored based on the data to be stored information; The availability KY of the data to be stored is calculated based on the information of the data to be stored; Substitute the access frequency FW, the update rate SX, the storage response delay YS, the required storage space size KJ, and the availability KY into the calculation formula of the data to be stored classification index to obtain the data to be stored classification index CCZ. The calculation formula is as follows: ； in, 、 、 are weight coefficients, which are 0.3, 0.4, and 0.3 respectively. The weight coefficients are used to measure the influence of different data to be stored on the classification of the data to be stored. The value of is 2.

72.

3. The cloud storage optimization system based on intelligent grading according to claim 2, characterized in that: The specific process of calculating the availability KY of the data to be stored based on the information of the data to be stored is as follows: Based on the information of the data to be stored, the MTTF (Mean Time to Failure to Use) and the MTTR (Mean Time to Repair) of the data to be stored are obtained, wherein the MTTR (Mean Time to Repair) of the data to be stored is the time required to repair the data to be stored based on the erasure coding technology; Calculate the availability KY of the data to be stored: 。 4. The cloud storage optimization system based on intelligent grading according to claim 1, characterized in that: The data to be stored is graded based on the CCZ, and the storage level corresponding to the data to be stored is determined. Specifically, the process includes the following: Load the grading index threshold of the data to be stored and ,in, and is pre-stored in the system, and Greater than ; If the CCZ of the data to be stored is less than , then the storage level corresponding to the data to be stored is the first-level storage level; If the CCZ of the data to be stored is greater than and less than , then the storage level corresponding to the data to be stored is the secondary storage level; If the CCZ of the data to be stored is greater than , then the storage level corresponding to the data to be stored is the third-level storage level.

5. The cloud storage optimization system based on intelligent grading according to claim 1, characterized in that: The cloud storage process based on the level of storage data includes the following steps: Step 1: Count the cache space of each node used to cache the data to be stored and gain , according to the unit space gain of each node Build a node priority queue in ascending order; Step 2: Get the probability that the total number of cache requests for data to be stored within time T is k and the data content size of the data to be stored , where the cache request for data to be stored within time T is recorded as a Poisson process; Step 3: Based on probability Build a content priority queue for data to be stored; Step 4: Determine whether the content priority queue of the data to be stored is empty. If so, the cloud storage ends. If not, the data to be stored at the top of the content priority queue of the data to be stored is used as the current cache object, and traverse in the order of the node priority queue to match the data content size of the current cache object. If the node exists, the current cache object is deployed on the node; if the node does not exist, the current cache object is removed from the content priority queue of the data to be stored and stored in the specified node, wherein the specified node is not in the node priority queue; Step 5: Repeat steps 2 to 4 until the content priority queue of the data to be stored is empty.

6. The cloud storage optimization system based on intelligent grading according to claim 5, characterized in that: The probability that the total number of cache requests for data to be stored within time T is k The specific process includes the following: ； in, It is the average number of requests to the cache for data to be stored per unit time.

7. The cloud storage optimization system based on intelligent grading according to claim 1, characterized in that: The system also includes a data storage update module. The workflow of the data storage update module is as follows: Initiate an update request: The user identifies the data that needs to be updated and initiates a data update request to the cloud server through the cloud storage service's API. The request includes the identifier of the data to be updated, the new data content or incremental data, and necessary authentication information. The cloud storage server updates the data in the storage system according to the user's request. The update process includes overwriting the original data, appending data to the end of the file, or updating the records in the database.

8. A cloud storage optimization method based on intelligent grading, applicable to a cloud storage optimization system based on intelligent grading according to any one of claims 1 to 7, characterized in that: Methods include: Acquiring information about the data to be stored, wherein the information about the data to be stored includes an access frequency of the data to be stored, a freshness rate of the data to be stored, a storage response delay of the data to be stored, a storage space required for the data to be stored, and availability of the data to be stored; Generate a classification index CCZ of data to be stored based on information of data to be stored; The data to be stored is graded based on the classification index CCZ of the data to be stored, and the storage level corresponding to the data to be stored is determined; Cloud storage based on the level of storage data.

Images