Data cold and hot hierarchical adaptive storage optimization system and method based on artificial intelligence

Abstract

The invention discloses a data cold and hot stratification self-adaptive storage optimization system and method based on artificial intelligence. The method comprises the steps that storage time sequence information of each piece of storage data in storage equipment is collected; inputting the storage time sequence information into a cold and hot state prediction model, and predicting a cold and hot state of the storage data in future time; acquiring the access frequency of the stored data in a unit period; establishing an access frequency analysis set for the access frequencies in the unit period, and extracting access frequency characteristics; the access frequency characteristics are input into a stage data recognition model, a stage judgment result is obtained, and the stage judgment result comprises yes or no; judging whether to migrate the storage data or not according to the staged judgment result of the storage data and the predicted cold and hot state of the storage data in the future time, and if not, judging whether to perform backup or not; data migration or non-migration decisions are reasonably made, frequent and unnecessary data migration is avoided to the maximum extent, and therefore the storage operation and maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to an artificial intelligence-based data hot and cold tiered adaptive storage optimization system and method. Background Art

[0002] With the rapid development of information technology and the deepening of digital transformation, the amount of data is showing an unprecedented explosive growth. The storage and efficient management of massive data has become a major challenge that needs to be urgently addressed in the field of information technology.

[0003] Traditional data storage systems have exposed many shortcomings in responding to this challenge. First, they often unilaterally use data access frequency as the only criterion for dividing hot and cold data, ignoring other important attributes of the data itself such as novelty and privacy sensitivity, resulting in a lack of precision in hot and cold tiered storage. This "one-size-fits-all" storage strategy lacks flexibility and specificity, and cannot fully adapt to the differentiated needs brought about by different data characteristics.

[0004] Secondly, when making data migration decisions, existing technologies often ignore the possible periodic fluctuations in data access patterns. The access frequency of certain data may fluctuate periodically over time, but traditional methods have weak recognition capabilities for this and are prone to making inappropriate frequent migration decisions for such data, resulting in waste of storage resources and inefficiency.

[0005] More importantly, existing methods are obviously blind in data migration and lack a comprehensive assessment and weighing of the negative impacts of frequent and unnecessary migration. Large amounts of frequent data migration will not only occupy a large amount of storage and network resources, affecting overall storage performance, but will also significantly increase operation and maintenance costs. At the same time, frequent data transmission will also increase the risk of data being stolen or tampered with during transmission, posing a threat to data security and integrity. However, existing technologies do not pay enough attention to these negative impacts and are prone to falling into the dilemma of blind and frequent migration.

[0006] In general, when facing the challenges of the big data era, traditional data storage systems have major deficiencies in terms of hot and cold stratification criteria, periodic fluctuation identification, migration decisions, etc. They lack intelligent management capabilities to adaptively adjust storage strategies based on data characteristics, which brings many challenges to storage efficiency, cost and security. New innovative solutions are urgently needed for optimization and improvement.

[0007] Therefore, it is an urgent problem for technical personnel in this field to propose an artificial intelligence-based data hot and cold tiered adaptive storage optimization system and method to solve the difficulties existing in the prior art. Summary of the invention

[0008] The purpose of the present invention is to provide an artificial intelligence-based data hot and cold tiered adaptive storage optimization system and method to address the deficiencies in the background technology.

[0009] In order to achieve the above object, the present invention provides the following technical solution: a data hot and cold tiered adaptive storage optimization method based on artificial intelligence, comprising:

[0010] Collecting storage timing information of each stored data in the storage device, wherein the storage timing information includes access frequency, privacy type, data novelty index and cold and hot state, wherein the cold and hot state includes cold state and hot state;

[0011] Input the storage time series information into the hot and cold state prediction model to predict the hot and cold state of the stored data in the future;

[0012] The access frequency within a unit cycle of the collected stored data;

[0013] The access frequency within the unit period is used to establish an access frequency analysis set, and the access frequency features are extracted;

[0014] The access frequency feature is input into the phased data recognition model to obtain the phased judgment result, which includes yes or no;

[0015] According to the periodic judgment result of the stored data and the predicted hot and cold state of the stored data in the future, it is determined whether to migrate the stored data. If not, it is determined whether to back up the data.

[0016] Furthermore, the method for determining whether to migrate the stored data includes:

[0017] If the stored data is currently stored on a thermal data storage medium, and the thermal state of the stored data is predicted at a future time, and the staged judgment result is yes or no, then the stored data is not migrated;

[0018] If the stored data is currently stored on a hot data storage medium and the cold state of the stored data is predicted in the future, and the interim judgment result is yes, then the stored data is not migrated;

[0019] If the stored data is currently stored on a hot data storage medium, and the cold state of the stored data is predicted in the future, and the interim judgment result is negative, the stored data is migrated to a cold data storage medium;

[0020] If the stored data is currently stored on a cold data storage medium, and the cold state of the stored data is predicted in the future, and the periodic judgment result is no or, then the stored data is not migrated;

[0021] If the stored data is currently stored on a cold data storage medium, and the thermal state of the stored data is predicted in the future, and the interim judgment result is negative, the stored data is migrated to a hot data storage medium.

[0022] Furthermore, if migration is not performed, the method for determining whether to perform backup includes:

[0023] If the stored data is currently stored on a cold data storage medium and the hot state of the stored data is predicted at a future time, and the interim judgment result is yes, the stored data is backed up to a hot data storage medium, and the stored data access interface is switched to the hot data storage medium; when the stored data is predicted to be in a cold state at the next future time, the backup of the stored data on the hot data storage medium is deleted.

[0024] Furthermore, the data novelty index is generated based on a comprehensive analysis of the inverse of the difference between the storage data collection time and the current time, the mean similarity between the storage data and other data, and the storage data type diversity index.

[0025] Furthermore, the storage data type diversity index uses entropy to measure the diversity of categories in the storage data.

[0026] Furthermore, the calculation method of the data novelty index is as follows:

[0027] NS=ω1×TL+ω2×DT+ω2×VY;

[0028] TL is the inverse of the difference between the storage data acquisition time and the current time; DT is the mean similarity between the storage data and other data; VY is the storage data type diversity index; ω1, ω2, ω2 are weight coefficients, which are non-negative numbers, and the sum of the weight coefficients is 1;

[0029] The TL formula is expressed as follows:

[0030] T1 represents the storage data acquisition time, and T2 represents the current time.

[0031] The DT formula is expressed as follows:

[0032] sx(A,B i ) represents the similarity score between the stored data and the i-th data in other data, and n is the number of other data;

[0033] Where a r , is the rth element of the stored data A, For other data B i The rth element in sx(A, B i) represents the Euclidean distance between the stored data and other data;

[0034] Furthermore, the VY calculation formula is as follows:

[0035]

[0036] In the formula, p g is the probability of category g appearing in the stored data, and G is the total number of categories contained in the stored data.

[0037] Furthermore, the method for constructing the cold and hot state prediction model includes:

[0038] Preset the prediction time step T, sliding step P2 and sliding window length P1; use the sliding window method to convert the stored time series information into multiple training samples, one training sample corresponds to one label, and constitutes a set of training data, P1 and P2 are both integers greater than 1;

[0039] The training data is used as the input of the hot and cold state prediction model, and the hot and cold state of the stored data after the predicted time step T is used as the output. The hot and cold state of the actual stored data in the future time after each time step T is used as the prediction target, and the prediction accuracy is used as the training target. The hot and cold state prediction model is trained to obtain a hot and cold state prediction model that meets the prediction accuracy. The hot and cold state prediction model is a temporal convolutional network, a long short-term memory network, or a gated recurrent unit.

[0040] Furthermore, the training method of the phased data recognition model includes:

[0041] Collect V groups of identification data, where V is an integer greater than 1, and the identification data includes access frequency characteristics and stage-by-stage judgment results corresponding to the access frequency characteristics, and the access frequency characteristics include access frequency change time series data, frequency mean, and frequency standard deviation;

[0042] The vector of each group of identification data is used as the input of the stage data identification model. The stage data identification model takes the stage judgment result corresponding to each group of access frequency feature prediction as the output, and takes the stage judgment result actually corresponding to each group of access frequency features as the prediction target; minimizing the sum of the prediction errors of the stage judgment results corresponding to all access frequency features is used as the training target; the stage data identification model is trained until the sum of the prediction errors converges and the training is stopped. The stage data identification model is a regression prediction model.

[0043] Furthermore, the method for obtaining the access frequency change time series data includes:

[0044] The difference between adjacent access frequencies in the access frequency analysis set is divided by the access frequency collected earlier to obtain the access frequency change. Similarly, the access frequency changes of adjacent access frequencies are calculated in sequence. All access frequency changes constitute the access frequency change time series data.

[0045] Furthermore, the frequency mean is the mean of the access frequencies in the access frequency analysis set; and the frequency standard deviation is the standard deviation of the access frequencies in the access frequency analysis set.

[0046] The data hot and cold tiered adaptive storage optimization system based on artificial intelligence implements the data hot and cold tiered adaptive storage optimization method based on artificial intelligence. The system includes:

[0047] A first acquisition module is used to acquire storage timing information of each storage data in the storage device, wherein the storage timing information includes access frequency, privacy type, data novelty index and cold and hot state, wherein the cold and hot state includes cold state and hot state;

[0048] The first analysis module is used to input the storage time series information into the cold and hot state prediction model to predict the cold and hot state of the storage data in the future;

[0049] The second acquisition module is used to acquire the access frequency of the stored data within a unit period;

[0050] A feature extraction module is used to establish an access frequency analysis set based on the access frequency in a unit period and extract access frequency features;

[0051] The second analysis module inputs the access frequency characteristics into the phased data recognition model to obtain the phased judgment result, which includes yes or no;

[0052] The comprehensive analysis module determines whether to migrate the stored data based on the periodic judgment results of the stored data and the predicted hot and cold states of the stored data in the future, and determines whether to back up the stored data if migration is not performed.

[0053] The present invention provides the following technical effects and advantages of the data hot and cold tiered adaptive storage optimization system and method based on artificial intelligence:

[0054] By building an advanced hot and cold state prediction model, it is possible to accurately predict the hot and cold state of stored data in future time periods, laying the foundation for accurate hot and cold tiered storage of data, thereby greatly improving the utilization efficiency of storage resources; even more cleverly, a new evaluation indicator, the data novelty index, is introduced, which comprehensively considers multiple dimensions such as data collection time, similarity with other data, and data diversity, and can more accurately and comprehensively evaluate the importance and value of data, providing a more accurate reference basis for hot and cold state judgment.

[0055] Secondly, by analyzing the timing characteristics of access frequency, an efficient phased data recognition model is constructed, which can accurately identify the phased fluctuation characteristics in the data access pattern, thereby avoiding making inappropriate frequent migration decisions for data with periodic access frequency fluctuations.

[0056] By comprehensively predicting the future hot and cold states and the periodic judgment results, we can make reasonable decisions on whether to migrate or not migrate data, and minimize frequent and unnecessary data migration, thereby reducing storage operation and maintenance costs and ensuring data transmission security and consistency.

[0057] In general, the innovative solution proposed in this invention breaks through the traditional "one-size-fits-all" storage strategy, realizes the intelligent management of data storage, can adaptively adjust the storage strategy to adapt to different data characteristics, and maximize the utilization efficiency of storage resources. It is an important innovative achievement in data storage optimization in the big data era. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0059] Figure 1 A schematic diagram of a data hot and cold tiered adaptive storage optimization system based on artificial intelligence of the present invention;

[0060] Figure 2 This is a flow chart of the artificial intelligence-based data hot and cold tiered adaptive storage optimization method of the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element at the same time.

[0063] Example 1

[0064] like Figure 1 As shown, the artificial intelligence-based data hot and cold tiered adaptive storage optimization system of this embodiment includes wired and / or wireless connections.

[0065] The first acquisition module is used to collect storage timing information of each storage data in the storage device, where the storage timing information includes access frequency, privacy type, data novelty index and hot and cold states, where the hot and cold states include cold state and hot state.

[0066] Hot data corresponds to high access frequency, and vice versa. Privacy types include privacy category (non-public data) and non-privacy category (public data). Privacy category data is less frequently accessed because it requires additional security measures and access control, and is classified as cold data. Non-privacy category data is widely shared and used, so it is also necessary to jointly determine whether it is hot data based on access frequency, privacy type, data growth, and data novelty.

[0067] Data sets with a high data novelty index, that is, recently generated or updated data sets, are more likely to become hot data because they may contain new information that is valuable to decision makers. Whether they are hot data also needs to be judged based on access frequency, privacy type, and data growth rate. Relatively old or outdated data sets may be more likely to become cold data because they have lost their timeliness.

[0068] The data novelty index is generated by comprehensive analysis of the inverse of the difference between the storage data collection time and the current time, the average similarity between the storage data and other data, and the storage data type diversity index. The calculation method of the data novelty index is as follows:

[0069] NS=ω1×TL+ω2×DT+ω2×VY;

[0070] TL is the inverse of the difference between the storage data acquisition time and the current time, and the TL value is positively correlated with the NS value; DT is the mean similarity between the storage data and other data (referring to the storage data other than the current storage data), and the DT value is positively correlated with the NS value; VY is the storage data type diversity index, and the VY value is positively correlated with the NS value; ω1, ω2, ω2 are weight coefficients, which are non-negative numbers, and the sum of the weight coefficients is 1, which is used to balance the influence of each parameter on the novelty score.

[0071] The TL formula is expressed as follows:

[0072] T1 represents the storage data acquisition time, and T2 represents the current time.

[0073] The DT formula is expressed as follows:

[0074] sx(A,B i) represents the similarity score between the stored data and the i-th data in other data, and n is the number of other data. When calculating the similarity score, the stored data and other data need to be converted into vectors, that is, vectorization, which can be done through natural language processing (NLP). Common ones are bag-of-words model, TF-IDF (TermFrequency-Inverse Document Frequency), One-Hot Encoding, etc.

[0075] Where a r , is the rth element of the stored data A, For other data B i The rth element in sx(A,B i ) represents the Euclidean distance between the stored data and other data. The smaller the value, the closer the stored data is to the other data, and vice versa. The similarity between the stored data and other data can also be evaluated by using the Jacquard similarity and cosine similarity calculation.

[0076] VY uses entropy to measure the diversity of categories in the stored data. The calculation formula is as follows:

[0077]

[0078] In the formula, p g is the probability of category g appearing in the stored data, G is the total number of categories contained in the stored data, the higher the VY (entropy), the greater the diversity and the stronger the novelty of the data; if all elements in the stored data belong to the same category, then the entropy will be 0, indicating no diversity; if the category distribution is very uneven, for example, most elements belong to a few categories, then the entropy value will be lower, indicating lower diversity, and vice versa.

[0079] Examples of categories include:

[0080] Biological classification (kingdom, phylum, class, order, family, genus, species), software application classification (operating system, office software, image editing, video editing, audio production, games), data type classification (quantitative data, qualitative data, text data, image data, audio data, video data), economic industry classification (agricultural manufacturing, service industry, financial industry, information technology), etc., the classification of stored data contains categories can be achieved by natural language processing technology.

[0081] The first analysis module is used to input the storage time series information into the cold and hot state prediction model to predict the cold and hot state of the storage data in the future.

[0082] The construction method of the cold and hot state prediction model includes:

[0083] Preset the prediction time step T, sliding step P2 and sliding window length P1; use the sliding window method to convert the stored time series information into multiple training samples, one training sample corresponds to one label, and constitutes a set of training data, P1 and P2 are both integers greater than 1;

[0084] The first set of training samples is X1, X1 = [T1, PV1, YS1, NS1], the hot and cold state corresponding to X1 is Y1; T1 is the timestamp in the first set of training samples, PV1 is the access frequency in the first set of training samples, YS1 is the privacy type in the first set of training samples, and NS1 is the data novelty index in the first set of training samples;

[0085] The training data is used as the input of the hot and cold state prediction model, and the hot and cold state of the stored data after the predicted time step T is used as the output. The hot and cold state of the actual stored data in the future time after each time step T is used as the prediction target, and the prediction accuracy is used as the training target. The hot and cold state prediction model is trained to obtain a hot and cold state prediction model that meets the prediction accuracy, that is, the hot and cold state prediction model is completed; the hot and cold state prediction model is an autoregressive integrated moving average model, a temporal convolutional network (TCN), a long short-term memory network (LSTM) or a gated recurrent unit (GRU).

[0086] The second acquisition module is used to acquire the access frequency of the stored data within a unit period.

[0087] The feature extraction module is used to establish an access frequency analysis set based on the access frequency in a unit period and extract access frequency features, wherein the access frequency features include access frequency change time series data, frequency mean and frequency standard deviation.

[0088] Methods for obtaining access frequency change timing data include:

[0089] The difference between adjacent access frequencies in the access frequency analysis set is divided by the access frequency before the collection time to obtain the access frequency change. Similarly, the access frequency changes of adjacent access frequencies are calculated in sequence. All access frequency changes constitute the access frequency change time series data. The access frequency change time series data reflects the volatility or periodicity of data access. The periodic access change means that the access frequency in different time periods is significantly different, which can be identified and quantified through the access frequency change time series data.

[0090] The frequency mean is the mean of the access frequencies in the access frequency analysis set; the frequency standard deviation is the standard deviation of the access frequencies in the access frequency analysis set; the frequency mean provides the average level of access frequency and is an overall measure that cannot provide information about the time distribution of access frequency changes; the frequency standard deviation measures the fluctuation or dispersion of access frequency and reflects the consistency of data access; a high standard deviation indicates that the access frequency has large fluctuations and there may be different access stages or cycles. The standard deviation is directly related to the data of staged access changes and can reveal the magnitude and frequency of access frequency changes.

[0091] The access frequency change time series data, frequency mean and frequency standard deviation are complementary metrics that together provide a comprehensive understanding of data access patterns and help identify and analyze periodic access changes. Through these metrics, we can better understand the dynamic characteristics of data access and make more accurate data migration decisions accordingly.

[0092] The second analysis module inputs the access frequency characteristics into the phased data recognition model to obtain the phased judgment result, which includes yes or no.

[0093] The training methods of the phased data recognition model include:

[0094] Collect V groups of identification data, where V is an integer greater than 1. The identification data includes access frequency characteristics and the periodic judgment results corresponding to the access frequency characteristics.

[0095] The vector of each group of identification data is used as the input of the stage data identification model, and the stage data identification model takes the stage judgment result corresponding to each group of access frequency feature prediction as the output, and takes the stage judgment result actually corresponding to each group of access frequency features as the prediction target; minimizing the sum of the prediction errors of the stage judgment results corresponding to all access frequency features is used as the training target; the stage data identification model is trained until the sum of the prediction errors converges and the training is stopped. The stage data identification model is a regression prediction model or other suitable machine learning model, which is not specifically limited here.

[0096] The calculation formula of prediction error is Z V =(δ V -μ V ) 2 , where Z V is the prediction error, V is the group number of the feature vector of the access frequency feature, δ V is the actual periodic judgment result corresponding to the access frequency feature of group V, μ V The corresponding interim judgment results for the visit frequency feature prediction of group V.

[0097] The comprehensive analysis module determines whether to migrate the stored data based on the periodic judgment results of the stored data and the predicted hot and cold states of the stored data in the future, and determines whether to back up the stored data if migration is not performed.

[0098] The method for determining whether to migrate the stored data includes:

[0099] If the stored data is currently stored on a hot data storage medium, and the thermal state of the stored data is predicted in the future, and the staged judgment result is yes or no, then the stored data is not migrated. Cold data storage media are such as HDDs, and hot data storage media are such as SSDs.

[0100] If the stored data is currently stored on a hot data storage medium, and the cold state of the stored data is predicted in the future, and the stage judgment result is yes, it means that the stored data is changing in stages, that is, the access frequency changes with the cycle, and the change is large. At the next time of the predicted future time, the data is likely to become hot again. At this time, the stored data is not migrated; avoid frequent migration of the stored data causing storage device performance degradation (because data needs to be copied and transmitted during the migration process, which will occupy network bandwidth and storage resources), increased storage costs (data migration involves the use of storage devices, network bandwidth and migration tools. Frequent migration does not bring expected cost savings, but increases overall expenses due to the cost of migration itself), data consistency and integrity risks (during the migration process, data will face consistency and integrity risks, especially when failures or interruptions occur during the migration process), data security issues (data will increase the risk of being exposed to an unsafe network environment during the migration process, increasing the risk of data leakage or tampering), increased management complexity (frequent data migration will increase the complexity of storage management, and more management work is required to track data location and status), etc.

[0101] If the stored data is currently stored on a hot data storage medium, and the cold state of the stored data is predicted in the future, and the stage-by-stage judgment result is no, it means that the stored data is not stage-changing data, that is, the access frequency changes with the cycle, and the change is small. At this time, the stored data is migrated to the cold data storage medium.

[0102] If the stored data is currently stored on a cold data storage medium, and the cold state of the stored data is predicted in the future, and the interim judgment result is no or yes, the stored data is not migrated.

[0103] If the stored data is currently stored on a cold data storage medium, and the thermal state of the stored data is predicted in the future, and the interim judgment result is negative, the stored data is migrated to a hot data storage medium.

[0104] If the stored data is currently stored on a cold data storage medium and the hot state of the stored data is predicted at a future time, and the interim judgment result is yes, the stored data is backed up to a hot data storage medium, and the stored data access interface is switched to the hot data storage medium; when the stored data is predicted to be in a cold state at a future time next time, the backup of the stored data on the hot data storage medium is deleted, and the stored data access interface is switched to the cold data storage medium. This avoids frequent migration of such stored data whose access frequency varies greatly with the periodic changes, causing adverse effects, and can also meet the user's access requirements for the stored data.

[0105] This embodiment proposes an innovative artificial intelligence-based data hot and cold tiered adaptive storage optimization system, which aims to solve the challenges faced by traditional data storage systems.

[0106] By building an advanced hot and cold state prediction model, the system can accurately predict the hot and cold state of stored data in future time periods, laying the foundation for accurate hot and cold tiered storage of data, thereby greatly improving the utilization efficiency of storage resources; even more cleverly, it introduces a new evaluation indicator, the data novelty index, which comprehensively considers multiple dimensions such as data collection time, similarity with other data, and data diversity. It can more accurately and comprehensively evaluate the importance and value of data, and provide a more accurate reference for hot and cold state judgment.

[0107] Secondly, by analyzing the timing characteristics of access frequency, an efficient phased data recognition model is constructed, which can accurately identify the phased fluctuation characteristics in the data access pattern, thereby avoiding making inappropriate frequent migration decisions for data with periodic access frequency fluctuations.

[0108] By comprehensively predicting the future hot and cold states and the periodic judgment results, we can make reasonable decisions on whether to migrate or not migrate data, and minimize frequent and unnecessary data migration, thereby reducing storage operation and maintenance costs and ensuring data transmission security and consistency.

[0109] In general, the innovative solution proposed in this embodiment breaks through the traditional "one-size-fits-all" storage strategy, realizes the intelligent management of data storage, can adaptively adjust the storage strategy to adapt to different data characteristics, and maximize the utilization efficiency of storage resources. It is an important innovative achievement in data storage optimization in the big data era.

[0110] Example 2

[0111] See also Figure 2 As shown, this embodiment provides an artificial intelligence-based data hot and cold tiered adaptive storage optimization method, including:

[0112] Collecting storage timing information of each stored data in the storage device, wherein the storage timing information includes access frequency, privacy type, data novelty index and cold and hot state, wherein the cold and hot state includes cold state and hot state;

[0113] Input the storage time series information into the hot and cold state prediction model to predict the hot and cold state of the stored data in the future;

[0114] The access frequency within a unit cycle of the collected stored data;

[0115] The access frequency within the unit period is used to establish an access frequency analysis set, and the access frequency features are extracted;

[0116] Input the access frequency feature into the stage data recognition model to obtain the stage judgment result, which includes yes or no;

[0117] According to the periodic judgment result of the stored data and the predicted hot and cold state of the stored data in the future, it is determined whether to migrate the stored data. If not, it is determined whether to back up the data.

[0118] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data hot and cold tiered adaptive storage optimization method based on artificial intelligence, characterized in that: include: Collecting storage timing information of each stored data in the storage device, wherein the storage timing information includes access frequency, privacy type, data novelty index and cold and hot state, wherein the cold and hot state includes cold state and hot state; Input the storage time series information into the hot and cold state prediction model to predict the hot and cold state of the stored data in the future; The access frequency within a unit cycle of the collected stored data; The access frequency within the unit period is used to establish an access frequency analysis set, and the access frequency features are extracted; Input the access frequency feature into the stage data recognition model to obtain the stage judgment result, which includes yes or no; According to the periodic judgment result of the stored data and the predicted hot and cold state of the stored data in the future, it is determined whether to migrate the stored data. If not, it is determined whether to back up the data.

2. The artificial intelligence-based data hot and cold tiered adaptive storage optimization method according to claim 1 is characterized in that: The method for determining whether to migrate the stored data includes: If the stored data is currently stored on a thermal data storage medium, and the thermal state of the stored data is predicted at a future time, and the staged judgment result is yes or no, then the stored data is not migrated; If the stored data is currently stored on a hot data storage medium and the cold state of the stored data is predicted in the future, and the interim judgment result is yes, then the stored data is not migrated; If the stored data is currently stored on a hot data storage medium, and the cold state of the stored data is predicted in the future, and the interim judgment result is negative, the stored data is migrated to a cold data storage medium; If the stored data is currently stored on a cold data storage medium, and the cold state of the stored data is predicted in the future, and the periodic judgment result is no or, then the stored data is not migrated; If the stored data is currently stored on a cold data storage medium, and the thermal state of the stored data is predicted in the future, and the interim judgment result is negative, the stored data is migrated to a hot data storage medium.

3. The artificial intelligence-based data hot and cold tiered adaptive storage optimization method according to claim 2 is characterized in that: If migration is not performed, the methods for determining whether to perform backup include: If the stored data is currently stored on a cold data storage medium and the hot state of the stored data is predicted at a future time, and the interim judgment result is yes, the stored data is backed up to a hot data storage medium, and the stored data access interface is switched to the hot data storage medium; when the stored data is predicted to be in a cold state at the next future time, the backup of the stored data on the hot data storage medium is deleted, and the stored data access interface is switched to the cold data storage medium.

4. The method for adaptive storage optimization of data hot and cold tiers based on artificial intelligence according to claim 1 is characterized in that: The data novelty index is generated based on a comprehensive analysis of the inverse of the difference between the storage data collection time and the current time, the mean similarity between the storage data and other data, and the storage data type diversity index.

5. The method for adaptive storage optimization of data hot and cold tiers based on artificial intelligence according to claim 1 is characterized in that: The storage data type diversity index uses entropy to measure the diversity of categories in the stored data.

6. The artificial intelligence-based data hot and cold tiered adaptive storage optimization method according to claim 2 is characterized in that: The method for constructing the cold and hot state prediction model includes: Preset the prediction time step T, sliding step P2 and sliding window length P1; use the sliding window method to convert the stored time series information into multiple training samples, one training sample corresponds to one label, and constitutes a set of training data, P1 and P2 are both integers greater than 1; The training data is used as the input of the hot and cold state prediction model, and the hot and cold state of the stored data after the predicted time step T is used as the output. The hot and cold state of the actual stored data in the future time after each time step T is used as the prediction target, and the prediction accuracy is used as the training target. The hot and cold state prediction model is trained to obtain a hot and cold state prediction model that meets the prediction accuracy. The hot and cold state prediction model is a temporal convolutional network, a long short-term memory network, or a gated recurrent unit.

7. The artificial intelligence-based data hot and cold tiered adaptive storage optimization method according to claim 6 is characterized in that: The training methods of the phased data recognition model include: Collect V groups of identification data, where V is an integer greater than 1, and the identification data includes access frequency characteristics and stage-by-stage judgment results corresponding to the access frequency characteristics, and the access frequency characteristics include access frequency change time series data, frequency mean, and frequency standard deviation; The vector of each group of identification data is used as the input of the stage data identification model. The stage data identification model takes the stage judgment result corresponding to each group of access frequency feature prediction as the output, and takes the stage judgment result actually corresponding to each group of access frequency features as the prediction target; minimizing the sum of the prediction errors of the stage judgment results corresponding to all access frequency features is used as the training target; the stage data identification model is trained until the sum of the prediction errors converges and the training is stopped. The stage data identification model is a regression prediction model.

8. The artificial intelligence-based data hot and cold tiered adaptive storage optimization method according to claim 7 is characterized in that: The method for obtaining the access frequency change time series data includes: The difference between adjacent access frequencies in the access frequency analysis set is divided by the access frequency collected earlier to obtain the access frequency change. Similarly, the access frequency changes of adjacent access frequencies are calculated in sequence. All access frequency changes constitute the access frequency change time series data.

9. The method for optimizing data hot and cold tiered adaptive storage based on artificial intelligence according to claim 7 is characterized in that: The frequency mean is the mean of the access frequencies in the access frequency analysis set; the frequency standard deviation is the standard deviation of the access frequencies in the access frequency analysis set.

10. The data hot and cold tiered adaptive storage optimization system based on artificial intelligence is characterized by: The system for implementing the artificial intelligence-based data hot and cold tiered adaptive storage optimization method according to any one of claims 1 to 9 includes: A first acquisition module is used to acquire storage timing information of each storage data in the storage device, wherein the storage timing information includes access frequency, privacy type, data novelty index and cold and hot state, wherein the cold and hot state includes cold state and hot state; The first analysis module is used to input the storage time series information into the cold and hot state prediction model to predict the cold and hot state of the storage data in the future; The second acquisition module is used to acquire the access frequency of the stored data within a unit period; A feature extraction module is used to establish an access frequency analysis set based on the access frequency in a unit period and extract access frequency features; The second analysis module inputs the access frequency characteristics into the phased data recognition model to obtain the phased judgment result, which includes yes or no; The comprehensive analysis module determines whether to migrate the stored data based on the periodic judgment results of the stored data and the predicted hot and cold states of the stored data in the future, and determines whether to back up the stored data if migration is not performed.

Images