Method and system for automatic data placement

Abstract

Method and system for automatic data placement. Method for automatically placing data within a storage system (40) having different types of storage media according to at least one data category, comprising at least the step of classifying the data according to their category, this classification being carried out by an online machine learning technique using information relating to said data. Figure for the abstract: Fig. 1

Description

Title of the invention: Method and system for automatic data placement technical field

[0001] The present invention relates to the field of data placement for storage systems, and more particularly to a method and system for automatic data placement. Previous technique

[0002] A data storage system can have different types of storage media. Each medium can have particular characteristics in terms of cost, access speed, or storage capacity. These systems can, for example, use SSDs (Solid State Drives), NVMe (Non-Volatile Memory Express), HDDs (Hard Disk Drives), optical discs, magnetic tapes, etc.

[0003] By "data placement" we mean the association or assignment of data to a storage medium.

[0004] In storage systems, the data temperature, which refers to the frequency of access to the data, is often used to distinguish hot data, which is accessed very frequently, from cold data, which is accessed infrequently.

[0005] Hot data can, for example, be placed on low-latency storage media such as SSDs and NVMe drives, which are often expensive. Cold data can be backed up on slower, but inexpensive, storage media such as HDDs, Blu-ray discs, or magnetic tapes.

[0006] Most data storage systems support migration solutions from one type of access (hot) to another (cold). This entails the consumption of significant resources (moving data from one location to another, requiring significant network and data processing resources), and requires data control tasks, as well as manual configuration by system administrators.

[0007] Data storage systems use different techniques that allow data to be moved from one storage medium to another (for example from high-cost storage media to low-cost media).

[0008] With these techniques, the relocation, if automatic, is based on static policies configured by the system administrator. An administrator can define relocation policies for each data item or group of data. The policy can include or exclude data of a certain type or based on their age or their size.

[0009] Recent predictions regarding stored data volumes forecast a global production of 163 zettabytes per year by 2025 (source: IDC). Faced with these colossal volumes of data to be stored, the current trend in the storage industry is to use machine learning and artificial intelligence (AI) techniques to develop learning methods that facilitate the automation of management tasks. Today, these techniques are applied for anomaly detection, predictive analytics (capacity forecasting), administrative support, disk failure detection, virtual machine migration, among other things.

[0010] In the article by Cheng, Peng, et al. "Optimizing Data Placement on Hierarchical Storage Architecture via Machine Learning", IFIP International Conference on Network and Parallel Computing, Springer, Cham, 2019., machine learning algorithms are used to perform optimal data placement based on the characteristics of a series of input commands ("workflow") to optimize system performance in terms of disk read / write.

[0011] This work, however, was developed for Big Data, based on the characteristics of command series, and aims to optimize task execution time and system performance in terms of input / output. The Gradient Boosting algorithm with the CART (Classification and Regression Tree) method is used to train the prediction / classification model.

[0012] The data placement in this publication does not consider the characteristics of the data but those of order series, without taking into account the relocation of the data and uses "Offline" learning algorithms and not "Online".

[0013] The authors Kale, RV, et al. in “A Practicable Machine Learning Solution for Security-Cognizant Data Placement on Cloud Platforms”, in handbook of computer networks and cyber security, Springer, used neural network-based data placement to ensure data availability in the event of security threats or cyberattacks.

[0014] A technique based on neural networks was also used by Ren, J., et al. "Archivist: a machine learning assisted data placement mechanism for hybrid storage systems", IEEE 37th International Conference on Computer Design (ICCD), to assist data placement and reduce file access latency. Their solution relies on estimating the file access model and using neural networks offline. Such a method cannot detect abrupt changes in data classification. Since the learning is performed offline, saving large volumes of data is possible. necessary to train the algorithms. Description of the invention

[0015] There is therefore a need to improve the processes of automatic data placement, particularly in terms of reducing storage costs. Automatic placement process

[0016] The invention aims to meet this objective and has as its object, according to one of its aspects, a method of automatic placement of data within a storage system having different types of storage media, according to at least one category of data, comprising at least the step of classifying the data according to their category, this classification being carried out by an online machine learning technique using information collected relating to said data.

[0017] By "data placement" we mean the association or assignment of data to a storage medium.

[0018] The invention is adapted to the various changes in the storage system, requires less computing power compared to methods based on neural networks and consumes few storage resources.

[0019] The invention reduces storage costs by providing proactive and appropriate data placement as soon as data enters the storage system, based on both its characteristics and its previously learned category. This avoids the need for subsequent migrations, thus reducing costs in terms of bandwidth and computing resources.

[0020] Optimizing storage infrastructure costs would allow storage providers to offer services at competitive rates with an improved quality of user experience (optimized data access times).

[0021] Online machine learning, operating on the fly, offers greater flexibility in that the administrator is relieved of manual and tedious configuration tasks. More dynamic behavior is possible (compared to static configurations).

[0022] In addition, online machine learning makes it possible to optimize relocation costs and improve response times.

[0023] The invention can be seen as an additional means of dealing with the complexity of storage management systems, a complexity accentuated by the variety of types of storage media deployed today, the varied needs (low latency, fast access, high throughput, etc.) of the storage services to be offered and the large volumes of data to be managed. Data category

[0024] The data can be files.

[0025] Preferably, said at least one data category is defined by the characteristics of said data, in particular the frequency of access to the data.

[0026] By correctly placing the data in the appropriate spaces, frequently accessed data will have fast access times.

[0027] In one embodiment, said at least one data category is the type of storage medium.

[0028] In another embodiment, said at least one data category is a storage service. This can be, for example, online storage, network attached storage (NAS: Network Attached Storage), or software defined storage (SDS: Software Defined Storage).

[0029] In another embodiment, said at least one data category is the storage location. Machine learning technique

[0030] Preferably, the online machine learning technique is based on online decision trees, and is in particular chosen from: the online random forest technique, a Hoeffding tree technique, and a stochastic gradient tree technique. The online random forest technique is suitable for solving multi-class classification problems, making it possible to predict the category to which the data belong.

[0031] The Online Random Forest technique has the advantage of consuming few CPU resources. It also offers high accuracy while providing faster execution compared to other supervised techniques. Furthermore, Online Random Forest is a simple and robust algorithm to implement, and it can handle unbalanced data. It can adapt to the class imbalance that can be encountered in a distributed storage system, which ensures greater accuracy compared to other machine learning models. For example, in two-tier hot / cold distributed storage systems, it is common to have more cold data than hot data. This imbalance between the proportion of hot and cold data can affect the machine learning model training process, leading to biased learning and erroneous predictions.

[0032] In addition, the "Online Random Forest" technique can be made parallel, which makes it possible to distribute the computational needs for the construction of the machine learning model between several nodes, the process according to the invention then being able to operate without affecting the main processes.

[0033] Moreover, this technique is interpretable, meaning that the machine learning model can be used not only for prediction, but also to reveal patterns hidden in the data. These patterns can be related to a The data management team can assist with monitoring and future actions without requiring significant prior configuration. Furthermore, Online Random Forest does not require extensive configuration, unlike gradient boosting or neural network-based methods, which necessitate a preliminary setup step.

[0034] The use of other online machine learning techniques based on decision trees is also possible, such as, for example, Hoeffding Tree techniques.

[0035] The Hoeffding Tree Classifier and the Hoeffding Tree Regressor are two other online tree algorithms that can be used within the framework of the method according to the invention. They use the Hoeffding limit to control split decisions and rely on the variance reduction calculation in the target space to decide among the split candidates. For these two techniques, learning is faster, but less accurate than the online random forest.

[0036] The Stochastic Gradient Trees technique can also be used. It is an online algorithm that minimizes a loss function instead of the variance of the target space. It has the advantage of requiring less memory than Hoeffding trees and online random forests; however, it cannot run in parallel. The history and characteristics of the data

[0037] Said information collected preferably includes at least metadata and a history of the data to be placed.

[0038] The data history includes information on the characteristics of said data extracted from the system trace files, also known as "log files". A "system log file" is understood to be a file containing a history of events that have occurred on the system.

[0039] The metadata may include at least the type of data, its size and information about the user.

[0040] The machine learning technique preferentially uses the history and characteristics of the data.

[0041] The characteristics of the data preferably include at least the type of data, its name, its size, its access path, its access frequency, and information about the user. Other steps in the process

[0042] In one embodiment, the process includes a preliminary step to classification, consisting of preparing the data to be placed by assigning them predefined categories based on information relating to said data.

[0043] The method preferably comprises the detection of a sudden change in category.

[0044] The method according to the invention is implemented by computer. By computer, we mean any electronic system, which may comprise a set of several machines, and which has computing capabilities. Typically, a computer comprises a processor, memory, a human-machine interface, for example a keyboard, a mouse or a touch screen, a display and a communication module, in particular Ethernet, Wi-Fi, Bluetooth® or using a mobile telephony technology, for example a technology operating with the GSM, GPRS, EDGE, LTE or UMTS protocols.

[0045] The method can be implemented in the form of software (software or computer program). Automatic placement system

[0046] The invention also relates, according to another aspect, to an automatic data placement system within a storage system, according to at least one data category, for implementing the method according to the invention. The placement system comprises a classification module configured to predict the data category using an online machine learning technique based on information relating to said data.

[0047] The system may include a data preparation module upstream of the classification module, configured to prepare the data to be placed by assigning them predefined categories based on information relating to said data.

[0048] The classification module preferably includes a sub-module for predicting a sudden change of category. Brief description of the drawings

[0049] The invention will be better understood upon reading the detailed description that follows, the non-limiting examples of its implementation, and upon examination of the accompanying drawing, on which:

[0050] [Fig.1] Fig.1 schematically represents an example of an automatic data placement system according to the invention. Detailed description

[0051] Figure 1 schematically illustrates an example of an automatic data placement system 1 according to the invention. This system 1 comprises a data preparation module 20 and a learning and classification module 30.

[0052] The data preparation module 20 includes a data extractor 21 configured to collect information relating to the data to be classified from a storage system 10.

[0053] The system 10 may include system trace files 11 (log files) and Data characteristics (including metadata such as data type, name, size, path, access frequency, user information, etc.) are considered. Other information may be included depending on the underlying storage architecture. Based on this information, the preparation module assigns predefined categories to the data. For example, the label "cold" is assigned to data that is rarely accessed, and the label "hot" is assigned to frequently accessed data.

[0054] The information on the data and their predefined categories (labels) then goes into the input of the learning and classification module 30. This module includes a learning engine 31 and a classifier 32, and its main objective is to be able to predict the category of a data based on its access activity.

[0055] The learning engine 31 uses an online machine learning technique, in the example described being the so-called "Online Random Forest" technique as presented above.

[0056] The classifier 32 is configured to predict the data access category based on the output of the learning engine 31.

[0057] The learning and classification module 30 may further include a sub-module 33 for predicting the stability of the data access model.

[0058] The concept of "drift" in machine learning refers to the abrupt change in the relationship between the input data and the output of the learning model. In the learning model of the present invention, a "drift detection" module can be integrated to identify abrupt changes in data categories.

[0059] Such a submodule 33 would be used for example to determine when data suddenly changes from a given category to a different category (in this case, data which was considered to be in low demand for a long time and which suddenly becomes in high demand by users).

[0060] Each data item is then placed in a storage medium 40 according to its predicted category, which can be "cold" or "hot". "Cold" data is placed in a slower access and less expensive medium 42, and "hot" data is placed in a high-performance and expensive medium 41.

[0061] The result of the automatic placement is returned via a feedback loop to the storage system 10 for an update of the log files 11 and metadata 12, with the aim of improving the accuracy of the storage system 10 and thus the quality of the machine learning.

[0062] HSM (Hierarchical Storage Management) systems, Tiering storage, object storage systems, file systems, Cloud Storage or Multi-cloud are systems where this invention can be applied.

[0063] The invention can be integrated into professional data storage or cloud software as well as management and monitoring systems for these systems.

[0064] It can also be applied, more generally, to any system where data must be placed in specific locations, provided that this data is characterized by specific attributes. Big Data can be considered another application.

[0065] Several players in the storage market could be potential customers for the application of this invention:

[0066] - storage and cloud service providers;

[0067] - companies that manage and optimize their own storage infrastructure;

[0068] - publishers of storage system software, or management software / monitoring of storage systems that could be positioned in an intermediate way and integrate the solution.

[0069] The invention is not limited to the embodiment described above.

[0070] Other machine learning techniques and other data categories are entirely conceivable.

Claims

Claims

1. Method for automatically placing data within a storage system (40) having different types of storage media, according to at least one category of data, comprising at least the step of classifying the data according to their category, this classification being carried out by an online automatic learning technique using information collected relating to said data and ensuring appropriate placement of the data upon their entry into the storage system, based both on their characteristics and on their previously learned category.

2. Method according to the preceding claim, said collected information comprising at least metadata (12) and a history (11) of the data to be placed.

3. Method according to one of the two preceding claims, said at least one category of data is defined by the characteristics of said data, in particular the frequency of access to the data.

4. A method according to any preceding claim, said at least one data category being the type of storage medium (40).

5. The method of claim 1, said at least one data category being a storage service (40).

6. The method of claim 1, said at least one category of data being the storage location.

7. Method according to any one of the preceding claims, the online machine learning technique being based on “Online Decision Trees”.

8. Method according to the preceding claim, the online machine learning technique based on decision trees being the “Online Random Forest” technique or a “Hoeffding Tree” technique or a “Stochastic Gradient Tree” technique.

9. Method according to any one of the preceding claims, the machine learning technique using the history and characteristics of the data. Method according to any one of the preceding claims, the characteristics of the data comprising at least the type of the data, its name, its size, its access path, its access frequency, the information on the user.

10. A method according to any preceding claim, comprising a step prior to classification, consisting of preparing the data to be placed by assigning them predefined categories based on information relating to said data (11, 12).

11. A method according to any preceding claim, comprising detecting an abrupt change of category.

12. System (1) for automatically placing data within a storage system (40) having different types of storage media according to at least one category of data, for implementing the method according to any one of the preceding claims, the placement system comprising a classification module (30) configured to predict the category of data using an online machine learning technique using information relating to said data.

13. System according to the preceding claim, comprising a data preparation module (20) upstream of the classification module, configured to prepare the data to be placed by assigning them predefined categories from the information relating to said data.

14. System according to claim 12 or 13, the classification module comprising a sub-module (33) for predicting a sudden change of category.

Images