An intelligent data classification and storage method and system based on deep learning

Abstract

The application relates to an intelligent data classification storage method and system based on deep learning, which comprises the following steps: obtaining an original data set, and obtaining a multidimensional data set after preprocessing; obtaining a frequency statistical set of data, and fitting the frequency statistical set by using a least square method to obtain an access frequency prediction value of each type of data; generating high-frequency or low-frequency data identification for corresponding data according to the comparison between the access frequency prediction value and a preset frequency threshold; and constructing a high-frequency data list and a low-frequency data list accordingly; obtaining a storage cost weight and an access delay index of a storage node according to a storage location index; calculating a storage efficiency score of each storage node; calculating an allocation proportion of each storage node, adjusting the allocation proportion through the storage efficiency score, and generating a multi-level storage allocation scheme; and after generating corresponding migration instructions according to the multi-level storage allocation scheme, migrating data to realize hierarchical classification storage of the data. The application can improve the utilization rate of system storage resources.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to an intelligent data classification and storage method and system based on deep learning. Background Technology

[0002] With the deepening development of the information age, data storage and management have become one of the core technologies supporting the digital transformation of various industries, and their importance is self-evident. Efficient data storage is not only related to system performance, but also directly affects the speed and cost optimization of data processing.

[0003] In one existing technology, the data storage system employs a single allocation strategy. Based on experience or historical data, storage media such as SSDs, HDDs, and tapes are divided into storage pools in a fixed proportion. Static rules are defined for each storage pool, such as SSDs storing only high-priority data or data with high historical access frequency, and HDDs storing low-priority data or data with low historical access frequency. However, priority only indicates the importance of data and does not correspond to the frequency of data access. Furthermore, the frequency of data access can change due to specific events or periodic tasks.

[0004] Therefore, defining only static rules will cause high-frequency access data to be mixed with low-frequency access data. For example, SSDs may be occupied by low-frequency data, while high-frequency data is forced to use HDDs, resulting in low storage resource utilization and difficulty in achieving fine-grained resource allocation. Summary of the Invention

[0005] This invention provides a deep learning-based intelligent data classification and storage method and system to solve the problems of low storage resource utilization and difficulty in achieving refined resource allocation in existing technologies.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides an intelligent data classification and storage method based on deep learning, comprising:

[0007] Obtain the original dataset and preprocess it to obtain a multidimensional dataset. Count the occurrence frequency of each type of label within a time window from the multidimensional dataset to obtain a frequency statistics set. Fit the frequency statistics set using the least squares method to obtain a predicted access frequency value for each type of data. Determine whether the predicted access frequency value exceeds a preset frequency threshold. If it does, generate a high-frequency data identifier for the corresponding data; otherwise, generate a low-frequency data identifier for the corresponding data. Based on the high-frequency and low-frequency data identifiers, record the data block size and storage location index to construct a high-frequency data list and a low-frequency data list. Access the configuration table of the storage node based on the storage location index to obtain the storage cost weight and access latency index of the storage node.

[0008] Based on the data block size and storage cost weight, calculate the storage efficiency score of each storage node; calculate the allocation ratio of each storage node based on the storage cost weight, the access latency index, and the access frequency prediction value; adjust the allocation ratio based on the storage efficiency score to obtain a multi-level storage allocation scheme.

[0009] According to the multi-level storage allocation scheme, the data identifiers in the high-frequency data list and the low-frequency data list are mapped to the corresponding storage nodes respectively, and migration instructions are generated. Data migration is then performed according to the migration instructions.

[0010] Preferably, the step of obtaining the original dataset and preprocessing the original dataset to obtain a multidimensional dataset includes:

[0011] Obtain the raw dataset containing the original timestamp sequence, the original access object identifier, and the original access type identifier from the storage system log;

[0012] The original timestamp sequence is converted into a sortable numerical form and arranged in ascending order to obtain the first dataset;

[0013] The uniqueness of the access object identifiers in the first dataset is verified by a hash algorithm. When a duplicate access object identifier is detected, a timestamp suffix is ​​added to generate a new unique identifier, thus obtaining the second dataset.

[0014] The access type identifiers in the second dataset are matched using a preset label mapping table. Unmatched access type identifiers are used to predict type labels using a Naive Bayes algorithm to obtain the third dataset.

[0015] By merging the third timestamp sequence, third access object identifier, and third access type identifier from the third dataset using MongoDB, a multidimensional dataset is obtained.

[0016] Preferably, the step of fitting the frequency statistics set using the least squares method to obtain the predicted access frequency value for each type of data includes:

[0017] Obtain the third timestamp sequence and the third access object identifier from the multidimensional dataset;

[0018] The periodic features of the third timestamp sequence are extracted by using a long short-term memory network, and the frequency statistics set is obtained by counting the number of times the third access object identifier appears within the time window.

[0019] A linear regression model is constructed, with the frequency statistics set as the target variable and the periodic features as the input features, and the optimal coefficients of the linear regression model are solved.

[0020] Based on the linear regression model, output the predicted access frequency for each data category.

[0021] Preferably, the step of calculating the storage efficiency score of each storage node based on the data block size and storage cost weight includes:

[0022] The data block size is normalized using a logarithmic function to obtain the normalized data block size;

[0023] Based on the normalized data block size, the storage cost weight, and the preset dynamic weight coefficient, the storage efficiency score of each storage node is calculated.

[0024] Preferably, the formula for calculating the storage efficiency score is:

[0025]

[0026] in, It is a storage efficiency score for each storage node; It is the size of the unnormalized data block; It is the normalized data block size; It is the storage cost weight, with a value of (0,1); and It is a dynamic weighting coefficient.

[0027] Preferably, the step of calculating the allocation ratio of each storage node based on the storage cost weight, the access latency index, and the access frequency prediction value includes:

[0028] The access latency metric and the predicted access frequency are normalized;

[0029] Calculate the comprehensive score for each type of data on each storage node, and sum the comprehensive scores to obtain the total node score for each storage node;

[0030] The sum of the node scores for all storage nodes is used to obtain the aggregate score.

[0031] The ratio of the sum of the node scores of each storage node to the total score is calculated and used as the allocation ratio for each storage node.

[0032] Preferably, the formula for calculating the comprehensive score is:

[0033]

[0034] in, For comprehensive scoring; This is the normalized predicted access frequency value; This is a normalized access latency metric. This is the storage cost weight, with a value of (0,1). , and These are the weighting coefficients.

[0035] Preferably, adjusting the allocation ratio based on the storage efficiency score to obtain a multi-tiered storage allocation scheme includes:

[0036] Determine whether the storage efficiency score is less than a preset score threshold. If so, reduce the allocation ratio of the corresponding storage node by the absolute value of the difference between the storage efficiency score and the preset score threshold.

[0037] If not, the allocation ratio of the corresponding storage node will be increased by the absolute value of the difference between the storage efficiency score and the preset score threshold.

[0038] A multi-tiered storage allocation scheme is generated based on the adjusted allocation ratio of each node.

[0039] Preferably, the step of mapping the data identifiers in the high-frequency data list and the low-frequency data list to corresponding storage nodes according to the multi-level storage allocation scheme and generating migration instructions, and performing data migration according to the migration instructions, includes:

[0040] A mapping tool is used to map the data identifiers of the high-frequency data list and the low-frequency data list to the corresponding storage nodes according to a multi-level storage allocation scheme, thereby generating a storage mapping set;

[0041] Based on the storage mapping set, and combining the original storage node address, data block size, and target storage node address, a migration instruction is generated;

[0042] After migrating the data according to the migration instructions, verify the data consistency.

[0043] Record migration logs and update the data storage location index.

[0044] Secondly, the present invention provides an intelligent data classification and storage system based on deep learning, comprising:

[0045] The data acquisition and processing module is used to acquire the original dataset, preprocess the original dataset, and obtain a multidimensional dataset.

[0046] The access frequency prediction module is used to obtain a frequency statistics set by counting the number of times each type of label appears within a time window from the multidimensional dataset, and then fit the frequency statistics set using the least squares method to obtain the access frequency prediction value for each type of data. The data identifier generation module is used to determine whether the access frequency prediction value exceeds a preset frequency threshold. If it does, a high-frequency data identifier is generated for the corresponding data; otherwise, a low-frequency data identifier is generated for the corresponding data. The data list construction module is used to record the data block size and storage location index based on the high-frequency data identifier and the low-frequency data identifier, and construct a high-frequency data list and a low-frequency data list. The node indicator acquisition module is used to access the configuration table of the storage node based on the storage location index to obtain the storage cost weight and access latency indicator of the storage node.

[0047] The storage efficiency score calculation module is used to calculate the storage efficiency score of each storage node based on the data block size and storage cost weight; the storage allocation scheme generation module is used to calculate the allocation ratio of each storage node based on the storage cost weight, the access latency index and the access frequency prediction value, and adjust the allocation ratio through the storage efficiency score to obtain a multi-level storage allocation scheme.

[0048] The migration instruction generation and execution module is used to map the data identifiers in the high-frequency data list and the low-frequency data list to the corresponding storage nodes according to the multi-level storage allocation scheme and generate migration instructions, and perform data migration according to the migration instructions.

[0049] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the deep learning-based intelligent data classification and storage method described in any one of the above.

[0050] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the deep learning-based intelligent data classification and storage method described above.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This invention calculates the allocation ratio of each storage node by using access latency metrics, storage cost weights, and predicted data access frequency, and adjusts the allocation based on storage efficiency scores to generate a multi-tiered storage allocation scheme. This dynamic allocation mechanism can dynamically allocate storage resources according to data access frequency and business needs, avoiding the mixing of high-frequency and low-frequency access data and improving storage resource utilization.

[0053] 2. This invention constructs high-frequency data lists and low-frequency data lists, and introduces storage efficiency scores to dynamically adjust storage resource allocation, generating a refined multi-level storage allocation scheme. This not only ensures that high-frequency data can be accessed quickly, but also makes full use of low-cost storage resources, thereby achieving optimal allocation of storage resources and effectively reducing storage costs.

[0054] 3. This invention obtains the original dataset of the storage system logs, preprocesses it to obtain a multidimensional dataset, and combines a Long Short-Term Memory (LSTM) network with the least squares method to fit the predicted access frequency value for each data category. The LSTM network can extract the periodic features of the timestamp sequence, and the least squares method is used to build the prediction model. By determining whether the predicted access frequency value exceeds a preset frequency threshold, high-frequency data identifiers and low-frequency data identifiers are generated, improving the accuracy of data classification. Attached Figure Description

[0055] Figure 1 This is a flowchart of an intelligent data classification and storage method based on deep learning according to the present invention.

[0056] Figure 2 This is a schematic diagram of a module of an intelligent data classification and storage system based on deep learning according to the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Reference Figure 1 The first embodiment of the present invention provides a flowchart of an intelligent data classification and storage method based on deep learning, including the following steps:

[0059] S11, Obtain the original dataset, preprocess the original dataset to obtain a multidimensional dataset;

[0060] S12, Count the occurrence frequency of each type of label within the time window from the multidimensional dataset to obtain a frequency statistics set. Fit the frequency statistics set using the least squares method to obtain a predicted access frequency value for each type of data. S13, Determine whether the predicted access frequency value exceeds a preset frequency threshold. If so, generate a high-frequency data identifier for the corresponding data; otherwise, generate a low-frequency data identifier for the corresponding data. S14, Based on the high-frequency and low-frequency data identifiers, record the data block size and storage location index to construct a high-frequency data list and a low-frequency data list. S15, Access the configuration table of the storage node based on the storage location index to obtain the storage cost weight and access latency index of the storage node.

[0061] S16, calculate the storage efficiency score of each storage node based on the data block size and storage cost weight; S17, calculate the allocation ratio of each storage node based on the storage cost weight, the access latency index and the access frequency prediction value, and adjust the allocation ratio through the storage efficiency score to obtain a multi-level storage allocation scheme.

[0062] S18, according to the multi-level storage allocation scheme, the data identifiers in the high-frequency data list and the low-frequency data list are mapped to the corresponding storage nodes respectively and migration instructions are generated, and data migration is performed according to the migration instructions.

[0063] In step S11, the original dataset is obtained, and the original dataset is preprocessed to obtain a multidimensional dataset, including:

[0064] Obtain the raw dataset containing the original timestamp sequence, the original access object identifier, and the original access type identifier from the storage system log;

[0065] The original timestamp sequence is converted into a sortable numerical form and arranged in ascending order to obtain the first dataset;

[0066] The uniqueness of the access object identifiers in the first dataset is verified by a hash algorithm. When a duplicate access object identifier is detected, a timestamp suffix is ​​added to generate a new unique identifier, thus obtaining the second dataset.

[0067] The access type identifiers in the second dataset are matched using a preset label mapping table. Unmatched access type identifiers are used to predict type labels using a Naive Bayes algorithm to obtain the third dataset.

[0068] By merging the third timestamp sequence, third access object identifier, and third access type identifier from the third dataset using MongoDB, a multidimensional dataset is obtained.

[0069] The process of obtaining the raw dataset from the storage system log can be achieved by reading the storage system log file using programming languages ​​such as Python or Java. The log data is then parsed according to its specific storage format, such as text, JSON, or XML, and relevant information such as timestamp sequences, access object identifiers, and access type identifiers is extracted. This relevant information can be organized into a table as the raw dataset.

[0070] It should be noted that the process of converting the timestamp sequence into a numerical form involves transforming the timestamp sequence into the total number of seconds since a fixed time, obtaining a numerical value, and then arranging all the values ​​corresponding to the timestamps in ascending order to obtain the first dataset. For example, the timestamp sequence 2023-10-01 10:00:10 can be converted into the total number of seconds since January 1, 1970, resulting in a value of 1696164010.

[0071] The uniqueness verification using a hash algorithm, which adds a unique timestamp suffix to duplicate access object identifiers, can be achieved through the following steps: Apply the MD5 hash algorithm to each access object identifier in the first dataset to generate a corresponding hash value for detecting identifier uniqueness; store the generated hash values ​​in a hash table; check whether each newly generated hash value already exists in the hash table. If so, generate a new unique identifier by adding the precise timestamp suffix of the current time, replace the original duplicate access object identifier with the generated new unique identifier, and update the dataset; if not, the data is proven to be unique, and no changes are made.

[0072] For example, if two object1s appear in the access object identifier, the unique access object identifier generated by adding the current timestamp suffix is ​​object1_1696164010, ensuring the uniqueness of the data and the accuracy of the categorized storage.

[0073] The process of matching access type identifiers using a preset label mapping table and predicting type labels not in the mapping table using the Naive Bayes algorithm is as follows: extract access type identifiers from the second dataset; match each access type identifier with an entry in the preset label mapping table; if a match is successful, replace the access type identifier with the corresponding standard type label; if the access type identifier is not in the mapping table, use the Naive Bayes algorithm to predict its type label and replace it, and then update the dataset.

[0074] The process of predicting type labels using the Naive Bayes algorithm is as follows: Collect labeled training data, including access type identifiers and their corresponding standard type labels; extract features of the access type identifiers, such as character length, included keywords, semantic information, etc.; use the Naive Bayes algorithm to build a prediction model and learn the probabilistic relationship between different features and standard labels; for unmatched access type identifiers, extract their features and use the trained prediction model to predict them to obtain the most likely standard label; replace the original unmatched access type identifiers with the predicted standard labels and update the dataset.

[0075] It is worth noting that the preset tag mapping table can be created through the following steps: collect all possible access type identifiers from the storage system logs, such as read, write, delete, update, etc., corresponding to read, write, delete, and update respectively; establish the mapping relationship between non-standard access type identifiers and standard tags, such as read_data mapping to read, write_data mapping to write, remove mapping to delete, and modify mapping to update; store the mapping relationship as a database table or other data structure to obtain the tag mapping table.

[0076] The specific implementation steps for merging the third dataset using MongoDB to obtain a multidimensional dataset are as follows: Connect to the MongoDB database using a MongoDB driver, such as PyMongo; insert each record in the third dataset into a MongoDB collection; design an aggregation pipeline using MongoDB's aggregation framework to merge the third timestamp sequence, the third access object identifier, and the third access type identifier into a multidimensional dataset.

[0077] In step S12, the frequency statistics set is fitted using the least squares method to obtain the predicted access frequency value for each type of data, including:

[0078] Obtain the third timestamp sequence and the third access object identifier from the multidimensional dataset;

[0079] The periodic features of the third timestamp sequence are extracted by using a long short-term memory network, and the frequency statistics set is obtained by counting the number of times the third access object identifier appears within the time window.

[0080] A linear regression model is constructed, with the frequency statistics set as the target variable and the periodic features as the input features, and the optimal coefficients of the linear regression model are solved.

[0081] Based on the linear regression model, output the predicted access frequency for each data category.

[0082] The process of obtaining the timestamp sequence and access object identifier from the multidimensional dataset requires connecting to a MongoDB database, querying the multidimensional dataset from the MongoDB collection, and then extracting the timestamp sequence and access object identifier.

[0083] It is worth noting that the extraction of periodic features from the third timestamp sequence via a Long Short-Term Memory (LSTM) network can be performed through the following steps: converting the timestamp sequence into a format suitable for LSTM model processing, such as converting the timestamps back to date and time format and setting them as indices; constructing an LSTM model using a TensorFlow or PyTorch deep learning framework; defining the model's input layer, LSTM layer, and output layer; training the LSTM model using time series data to learn the periodic trends in the sequence; selecting a suitable loss function and optimizer, such as MSE (mean squared error) and the Adam optimizer; and extracting periodic features from the hidden layers of the LSTM model.

[0084] The frequency statistics set obtained by counting the number of times the third access object identifier appears within the time window can be obtained using Pandas or other data processing tools. The data can be grouped by the third access object identifier, and the window can be moved over the entire time series using the sliding window technique. The number of times the third access object identifier appears within each window can be counted, and the statistical results can be stored as a DataFrame or other suitable data structure as the frequency statistics set.

[0085] The construction of the linear regression model, using the frequency statistics set as the target variable and the periodic features as input features, and solving for the optimal coefficients of the linear regression model, can be achieved through the following steps: normalizing or standardizing the input features and target variable; randomly initializing the weight coefficients and bias terms of the linear regression model; using the mean squared error as the loss function to measure the difference between the model's predicted value and the actual target variable; minimizing the loss function using the gradient descent algorithm, gradually adjusting the model parameters to minimize the error between the predicted and actual values; and using the trained model to predict the access frequency of each type of data.

[0086] The optimal coefficients of the linear regression model can be obtained by minimizing the sum of squared residuals between the predicted and actual values.

[0087] In step S13, it is determined whether the predicted access frequency value exceeds a preset frequency threshold. If so, a high-frequency data identifier is generated for the corresponding data; otherwise, a low-frequency data identifier is generated for the corresponding data.

[0088] The preset frequency threshold can be determined based on business needs and historical data analysis, such as selecting the median or average access frequency. For example, if the business has high real-time requirements for data access, a higher frequency threshold can be set, such as more than 10 accesses per hour being considered high-frequency data and less than 10 accesses per hour being considered low-frequency data. The high-frequency data identifier and the low-frequency data identifier can be a fixed string, such as "high" and "low," or other strings or tags that can distinguish between high and low.

[0089] In step S14, based on the high-frequency data identifier and the low-frequency data identifier, the data block size and storage location index are recorded to construct the high-frequency data list and the low-frequency data list.

[0090] The construction of the high-frequency data list and the low-frequency data list can be implemented through the following steps: create two empty lists to store relevant information of high-frequency data and low-frequency data respectively; traverse all generated high-frequency data identifiers and low-frequency data identifiers; add the corresponding data block size and storage location index to the corresponding list according to the type of data identifier to obtain the high-frequency data list and the low-frequency data list.

[0091] In step S15, the configuration table of the storage node is accessed according to the storage location index to obtain the storage cost weight and access latency index of the storage node.

[0092] It should be noted that the storage node configuration table contains the storage location index, storage node type, storage cost weight, and access latency metric for the corresponding data. This information can be used as the table header. For example, the corresponding information for a piece of data in the storage node configuration table is as follows: storage location index is POST1, storage node type is SSD, storage cost weight is 0.8, and access latency metric is 5ms.

[0093] In step S16, the storage efficiency score of each storage node is calculated based on the data block size and storage cost weight, including:

[0094] The data block size is normalized using a logarithmic function to obtain the normalized data block size;

[0095] The storage efficiency score of each storage node is calculated based on the normalized data block size, the storage cost weight, and the preset dynamic weight coefficient.

[0096] Specifically, the normalization of the data block size using a logarithmic function with base 10 is performed. During normalization, the unit of data block size must be unified to GB. To avoid mathematical errors in the calculation formula when the data block size is 0, the data block size needs to be increased by 1 to effectively handle zero-value data blocks.

[0097] It is worth noting that the formula for calculating the storage efficiency score is as follows:

[0098]

[0099] in, It is a storage efficiency score for each storage node; It is the size of the unnormalized data block; It is the normalized data block size; It is the storage cost weight, with a value of (0,1); and It is a dynamic weighting coefficient.

[0100] For example, the dynamic weighting coefficients can be defined based on the remaining capacity of the storage node; for instance, when the remaining SSD capacity is <30%, the dynamic weighting coefficients for data block size and storage cost weights are 0.6 and 0.4, respectively; when the remaining SSD capacity is ≥30%, the dynamic weighting coefficients for data block size and storage cost weights are 0.4 and 0.6, respectively.

[0101] It should be noted that taking the reciprocal of the logarithmically processed block size is to inversely adjust the impact of block size on storage efficiency. The smaller the block size, the larger the reciprocal, and the higher the score contribution, encouraging the storage of small blocks and optimizing space utilization; large blocks have lower reciprocal scores, preventing them from occupying valuable space on high-performance storage media, thereby reducing cost waste.

[0102] In step S17, the allocation ratio of each storage node is calculated based on the storage cost weight, the access latency index, and the access frequency prediction value. The allocation ratio is then adjusted using the storage efficiency score to obtain a multi-tiered storage allocation scheme, including:

[0103] The access latency metric and the predicted access frequency are normalized;

[0104] Calculate the comprehensive score for each type of data on each storage node, and sum the comprehensive scores to obtain the total node score for each storage node;

[0105] The sum of the node scores for all storage nodes is used to obtain the aggregate score.

[0106] The ratio of the sum of the node scores of each storage node to the total score is calculated and used as the allocation ratio for each storage node.

[0107] The normalization of the access latency index can be calculated by the ratio of the minimum value to the current value; the normalization of the access frequency prediction value is calculated by the ratio of the current value to the maximum value.

[0108] It is worth noting that the formula for calculating the comprehensive score is as follows:

[0109]

[0110] in, For comprehensive scoring, This is the normalized predicted access frequency value. This is a normalized access latency metric. The storage cost weight has a value of (0,1). , and These are the weighting coefficients, with default values ​​of 0.5, 0.3, and 0.2, which can be adjusted according to business needs.

[0111] The calculation of the allocation ratio of each storage node requires summing the comprehensive scores of each type of data on each storage node to obtain the total node score of each storage node, summing the total node scores of all storage nodes to obtain the total score of all storage nodes, and dividing the total node score of each storage node by the total score to obtain the allocation ratio of the corresponding storage node.

[0112] In one implementation, adjusting the allocation ratio based on the storage efficiency score to obtain a multi-tiered storage allocation scheme includes:

[0113] Determine whether the storage efficiency score is less than a preset score threshold. If so, reduce the allocation ratio of the corresponding storage node by the absolute value of the difference between the storage efficiency score and the preset score threshold.

[0114] If not, the allocation ratio of the corresponding storage node will be increased by the absolute value of the difference between the storage efficiency score and the preset score threshold.

[0115] A multi-tiered storage allocation scheme is generated based on the adjusted allocation ratio of each node.

[0116] It should be noted that the preset scoring threshold can be set according to the business needs of different storage nodes. For example, if the business requires a storage efficiency score of 0.8 or higher to meet performance requirements, the preset scoring threshold can be set to 0.8; for businesses with high real-time requirements, such as financial transaction systems and e-commerce shopping carts, a higher preset scoring threshold can be set, such as 0.7-0.9; while for businesses with relatively low real-time requirements, such as data analysis backend processing and archiving systems, a lower preset scoring threshold can be set, such as 0.3-0.5.

[0117] Specifically, the difference between a preset scoring threshold and the storage efficiency score of each storage node can be calculated; then the ratio of the difference to the preset scoring threshold can be calculated; then the ratio can be subtracted from 1 to obtain the adjustment result of the allocation ratio of the corresponding storage node; the adjustment result can be multiplied by the original allocation ratio of the corresponding storage node to obtain the adjusted allocation ratio; and a multi-level storage allocation scheme can be generated based on the adjusted allocation ratio.

[0118] For example, if node C has an allocation ratio of 70% and node D has an allocation ratio of 30%, and node D's storage efficiency score is only 0.6, which is lower than the set 0.8, then the absolute value of the difference between node D's storage efficiency score and the preset score threshold is calculated to be 25%. Therefore, the allocation ratio of node D is reduced by 25%, resulting in an adjusted allocation ratio of 22.5% for node D, while the corresponding allocation ratio of node C increases to 77.5%. This yields the adjusted allocation ratio set.

[0119] In step S18, according to the multi-level storage allocation scheme, the data identifiers in the high-frequency data list and the low-frequency data list are mapped to corresponding storage nodes respectively, and migration instructions are generated. Data migration is performed according to the migration instructions, including:

[0120] A mapping tool is used to map the data identifiers of the high-frequency data list and the low-frequency data list to the corresponding storage nodes according to a multi-level storage allocation scheme, thereby generating a storage mapping set;

[0121] Based on the storage mapping set, and combining the original storage node address, data block size, and target storage node address, a migration instruction is generated;

[0122] After migrating the data according to the migration instructions, verify the data consistency.

[0123] Record migration logs and update the data storage location index.

[0124] The process of using a mapping tool to map data identifiers to corresponding storage nodes according to a multi-level storage allocation scheme and generating a storage mapping set requires the following steps: loading a multi-level storage allocation scheme that specifies the storage nodes to which each data identifier should be migrated; traversing the high-frequency data list and the low-frequency data list; mapping each data identifier to the corresponding storage node; and saving all mapping relationships to a set to form a storage mapping set.

[0125] The process of generating a migration instruction based on the storage mapping set, combined with the original storage node address, data block size, and target storage node address, requires first parsing the storage mapping set to extract the target storage node address corresponding to each data representation; then, obtaining the current storage location of the data based on the storage location index; obtaining the data block size based on records in the high-frequency data list and the low-frequency data list; and finally, synthesizing the original storage node address, data block size, and target storage node address into a structured migration instruction. For example, the migration instruction must include the following fields: unique data identifier, original data storage node address, target storage node address, data block size, and migration priority. The data block size is used to estimate migration bandwidth, and the migration priority is distinguished based on the high-frequency and low-frequency data identifiers.

[0126] It should be noted that after data migration is performed according to the migration instruction, data consistency can be verified by regenerating the hash value of the migrated data and comparing it with the hash value of the data in the original storage address. If the comparison is consistent, the migration is considered successful; if the comparison is inconsistent, the migration is restarted. A migration log is recorded, and the storage location index is updated for the successfully migrated data, thus completing the hierarchical classification and storage of the data.

[0127] In summary, this invention calculates the allocation ratio of each storage node by acquiring the access latency index, storage cost weight, and predicted data access frequency of the storage nodes, and adjusts the allocation ratio based on the storage efficiency score to generate a multi-tiered storage allocation scheme. This dynamic allocation mechanism can dynamically allocate storage resources according to the data access frequency and business needs, avoiding the mixing of high-frequency access data with low-frequency access data and improving storage resource utilization.

[0128] Reference Figure 2 The second embodiment of the present invention provides an intelligent data classification and storage system based on deep learning, comprising:

[0129] The data acquisition and processing module is used to acquire the original dataset, preprocess the original dataset, and obtain a multidimensional dataset.

[0130] The access frequency prediction module is used to obtain a frequency statistics set by counting the number of times each type of label appears within a time window from the multidimensional dataset, and then fit the frequency statistics set using the least squares method to obtain the access frequency prediction value for each type of data. The data identifier generation module is used to determine whether the access frequency prediction value exceeds a preset frequency threshold. If it does, a high-frequency data identifier is generated for the corresponding data; otherwise, a low-frequency data identifier is generated for the corresponding data. The data list construction module is used to record the data block size and storage location index based on the high-frequency data identifier and the low-frequency data identifier, and construct a high-frequency data list and a low-frequency data list. The node indicator acquisition module is used to access the configuration table of the storage node based on the storage location index to obtain the storage cost weight and access latency indicator of the storage node.

[0131] The storage efficiency score calculation module is used to calculate the storage efficiency score of each storage node based on the data block size and storage cost weight; the storage allocation scheme generation module is used to calculate the allocation ratio of each storage node based on the storage cost weight, the access latency index and the access frequency prediction value, and adjust the allocation ratio through the storage efficiency score to obtain a multi-level storage allocation scheme.

[0132] The migration instruction generation and execution module is used to map the data identifiers in the high-frequency data list and the low-frequency data list to the corresponding storage nodes according to the multi-level storage allocation scheme and generate migration instructions, and perform data migration according to the migration instructions.

[0133] It should be noted that the intelligent data classification and storage system based on deep learning provided in this embodiment of the invention is used to execute all the process steps of the intelligent data classification and storage method based on deep learning in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0134] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a Moore-Neighbor tracking program. When the processor executes the computer program, it implements the steps described in the various embodiments of the AI-driven precision tool appearance dimension data detection method, for example... Figure 1 The step S11 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as the storage efficiency scoring calculation module.

[0135] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0136] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0137] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0138] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0139] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0140] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0141] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A deep learning-based intelligent data classification and storage method, characterized in that, include: Obtain the original dataset, preprocess the original dataset to obtain a multidimensional dataset; The frequency statistics set is obtained by counting the number of times each type of data appears in the time window from the multidimensional dataset. The least squares method is used to fit the frequency statistics set to obtain the predicted access frequency value for each type of data. Determine whether the predicted access frequency value exceeds a preset frequency threshold. If it does, generate a high-frequency data identifier for the corresponding data; otherwise, generate a low-frequency data identifier for the corresponding data. Based on the high-frequency data identifier and the low-frequency data identifier, record the data block size and storage location index, and construct a high-frequency data list and a low-frequency data list; Access the configuration table of the storage node based on the storage location index to obtain the storage cost weight and access latency index of the storage node; Based on the data block size and storage cost weight, calculate the storage efficiency score for each storage node; The allocation ratio of each storage node is calculated based on the storage cost weight, the access latency index, and the access frequency prediction value. The allocation ratio is then adjusted using the storage efficiency score to obtain a multi-level storage allocation scheme. According to the multi-level storage allocation scheme, the data identifiers in the high-frequency data list and the low-frequency data list are mapped to corresponding storage nodes respectively, and migration instructions are generated. Data migration is then performed according to the migration instructions. The step of calculating the storage efficiency score of each storage node based on the data block size and storage cost weight includes: The data block size is normalized using a logarithmic function to obtain the normalized data block size; Based on the normalized data block size, the storage cost weight, and the preset dynamic weight coefficient, the storage efficiency score of each storage node is calculated. The formula for calculating the storage efficiency score is as follows: in, It is a storage efficiency score for each storage node; It is the size of the unnormalized data block; It is the normalized data block size; It is the storage cost weight, with a value of (0,1); and It is a dynamic weighting coefficient, which is set according to the remaining capacity of the storage node; The step of adjusting the allocation ratio based on the storage efficiency score to obtain a multi-tiered storage allocation scheme includes: If the storage efficiency score is less than a preset score threshold, the allocation ratio of the corresponding storage node is reduced by the absolute value of the difference between the storage efficiency score and the preset score threshold; otherwise, the allocation ratio of the corresponding storage node is increased by the absolute value of the difference between the storage efficiency score and the preset score threshold. Based on the adjusted allocation ratio of each node, a multi-level storage allocation scheme is generated; The process involves calculating the difference between a preset scoring threshold and the storage efficiency score of each storage node; then calculating the ratio of the difference to the preset scoring threshold; subtracting the ratio from 1 to obtain the adjusted allocation ratio for the corresponding storage node; and multiplying the adjusted ratio by the original allocation ratio of the corresponding storage node to obtain the adjusted allocation ratio.

2. The intelligent data classification and storage method based on deep learning according to claim 1, characterized in that, The process of obtaining the original dataset and preprocessing the original dataset to obtain a multidimensional dataset includes: Obtain the raw dataset containing the original timestamp sequence, the original access object identifier, and the original access type identifier from the storage system log; The original timestamp sequence is converted into a sortable numerical form and arranged in ascending order to obtain the first dataset; The uniqueness of the access object identifiers in the first dataset is verified by a hash algorithm. When a duplicate access object identifier is detected, a timestamp suffix is ​​added to generate a new unique identifier, thus obtaining the second dataset. The access type identifiers in the second dataset are matched using a preset label mapping table. Unmatched access type identifiers are used to predict type labels using a Naive Bayes algorithm to obtain the third dataset. By merging the third timestamp sequence, third access object identifier, and third access type identifier from the third dataset using MongoDB, a multidimensional dataset is obtained.

3. The intelligent data classification and storage method based on deep learning according to claim 2, characterized in that, The process of fitting the frequency statistics set using the least squares method to obtain the predicted access frequency value for each type of data includes: Obtain the third timestamp sequence and the third access object identifier from the multidimensional dataset; The periodic features of the third timestamp sequence are extracted by using a long short-term memory network, and the frequency statistics set is obtained by counting the number of times the third access object identifier appears within the time window. A linear regression model is constructed, with the frequency statistics set as the target variable and the periodic features as the input features, and the optimal coefficients of the linear regression model are solved. Based on the linear regression model, output the predicted access frequency for each data category.

4. The intelligent data classification and storage method based on deep learning according to claim 1, characterized in that, The step of calculating the allocation ratio of each storage node based on the storage cost weight, the access latency index, and the access frequency prediction value includes: The access latency metric and the predicted access frequency are normalized; Calculate the comprehensive score for each type of data on each storage node, and sum the comprehensive scores to obtain the total node score for each storage node; The sum of the node scores for all storage nodes is used to obtain the aggregate score. The ratio of the sum of the node scores of each storage node to the total score is calculated and used as the allocation ratio for each storage node.

5. The intelligent data classification and storage method based on deep learning according to claim 4, characterized in that, The formula for calculating the comprehensive score is as follows: in, For comprehensive scoring; This is the normalized predicted access frequency value; This is a normalized access latency metric. This is the storage cost weight, with a value of (0,1). , and These are the weighting coefficients.

6. The intelligent data classification and storage method based on deep learning according to claim 1, characterized in that, The step of mapping data identifiers in the high-frequency data list and the low-frequency data list to corresponding storage nodes according to the multi-level storage allocation scheme and generating migration instructions, and performing data migration according to the migration instructions, includes: A mapping tool is used to map the data identifiers of the high-frequency data list and the low-frequency data list to the corresponding storage nodes according to a multi-level storage allocation scheme, thereby generating a storage mapping set; Based on the storage mapping set, and combining the original storage node address, data block size, and target storage node address, a migration instruction is generated; After migrating the data according to the migration instructions, verify the data consistency. Record migration logs and update the data storage location index.

7. A deep learning-based intelligent data classification and storage system, characterized in that, The method for implementing the deep learning-based intelligent data classification and storage method as described in any one of claims 1-6 includes: The data acquisition and processing module is used to acquire the original dataset, preprocess the original dataset, and obtain a multidimensional dataset. The access frequency prediction module is used to obtain a frequency statistics set by counting the number of times each type of label appears in the time window from the multidimensional dataset, and to fit the frequency statistics set by the least squares method to obtain the access frequency prediction value for each type of data. The data identifier generation module is used to determine whether the predicted access frequency value exceeds a preset frequency threshold. If it does, a high-frequency data identifier is generated for the corresponding data; otherwise, a low-frequency data identifier is generated for the corresponding data. The data list construction module is used to construct high-frequency data lists and low-frequency data lists by recording data block sizes and storage location indexes based on the high-frequency data identifiers and the low-frequency data identifiers. The node metric acquisition module is used to access the configuration table of the storage node according to the storage location index to obtain the storage cost weight and access latency metric of the storage node. The storage efficiency score calculation module is used to calculate the storage efficiency score of each storage node based on the data block size and storage cost weight. The storage allocation scheme generation module is used to calculate the allocation ratio of each storage node based on the storage cost weight, the access latency index and the access frequency prediction value, and adjust the allocation ratio through the storage efficiency score to obtain a multi-level storage allocation scheme. The migration instruction generation and execution module is used to map the data identifiers in the high-frequency data list and the low-frequency data list to the corresponding storage nodes according to the multi-level storage allocation scheme and generate migration instructions, and perform data migration according to the migration instructions.

Images