A storage control method and device based on big data

Abstract

The present invention relates to the field of big data storage technology, and in particular to a storage control method and device based on big data. The method comprises the following steps: obtaining the original data stream and application requirement information; performing data dynamic correlation analysis on the original data stream according to the application requirement information to obtain a dynamically updated data correlation map; generating feature vectors for the dynamically updated data correlation map to obtain data feature vectors; performing intelligent quality assessment on the data feature vectors to obtain data quality labels; performing timeliness assessment based on the data feature vectors and performing access heat assessment to obtain timeliness scores and access heat scores; and obtaining system resource load information. The present invention improves storage resource utilization, data access performance, and overall system efficiency by solving the limitations of traditional methods in static data management and resource allocation.

Description

Technical Field

[0001] The present invention relates to the field of big data storage technology, and in particular to a storage control method and device based on big data. Background Art

[0002] With the rapid development of technologies such as the Internet and the Internet of Things, the amount of data has exploded, and the data types have become increasingly diverse, including structured data, semi-structured data, and unstructured data. This has brought huge challenges to traditional storage control methods. In order to cope with the needs of massive data storage and management in a big data environment, various new storage control methods have emerged, such as distributed storage, cloud storage, and object storage. However, most of these methods still have some limitations, especially in static data management and resource allocation.

[0003] 1. Limitations of static data management:

[0004] Static tiering: Traditional storage systems usually statically divide data into different tiers based on the performance and cost of the storage media, such as storing hot data in SSDs and cold data in HDDs. This static tiering method cannot adapt to the dynamic changes in data access patterns. For example, some data that was initially accessed less frequently (cold data) becomes an access hotspot (hot data) over time, and static tiering cannot adjust the storage location of the data in time, resulting in reduced access performance. In addition, static tiering also has difficulty handling data with complex access patterns, such as some data that is sometimes hot data and sometimes cold data.

[0005] Static data layout: Traditional methods usually plan the physical storage location of data in advance without considering the access pattern of data. In a big data environment, the access pattern of data often changes dynamically. Static data layout is prone to data hotspot problems, that is, some storage nodes are overloaded while other node resources are idle, thereby reducing the overall performance and reliability of the system.

[0006] 2. Inflexible resource allocation:

[0007] Static resource pre-allocation: Traditional methods usually pre-allocate fixed resources for different storage tiers or storage nodes, such as bandwidth, IOPS, storage space, etc. This static resource allocation method cannot be adjusted according to the dynamic changes in data access patterns, resulting in resource waste or performance bottlenecks. For example, if the access volume of high-value data suddenly increases, and the pre-allocated resources are insufficient, it will lead to increased access latency and affect application performance.

[0008] Lack of refined control: Static resource allocation is usually coarse-grained and cannot provide refined control over different types of data. For example, some latency-sensitive applications require higher bandwidth and IOPS guarantees, but static resource allocation cannot meet this demand. Summary of the invention

[0009] Based on this, it is necessary to provide a storage control method and device based on big data to solve at least one of the above technical problems.

[0010] To achieve the above purpose, a storage control method based on big data includes the following steps:

[0011] Step S1: obtaining the original data stream and application requirement information; performing data dynamic correlation analysis on the original data stream according to the application requirement information to obtain a dynamically updated data correlation map; generating a feature vector for the dynamically updated data correlation map to obtain a data feature vector;

[0012] Step S2: Perform intelligent quality assessment on the data feature vector to obtain a data quality label; perform timeliness assessment based on the data feature vector and perform access heat assessment to obtain a timeliness score and an access heat score; obtain system resource load information; perform application scenario weight calculation based on the data feature vector and the system resource load information to obtain an application scenario weight; perform data value assessment based on the timeliness score, access heat score, application scenario weight and data quality label to obtain a data value score; perform data strategy classification processing based on the data value score to obtain data value level data;

[0013] Step S3: adaptively allocate resources according to the data value level data to obtain a resource allocation plan;

[0014] Step S4: Perform intelligent data cleaning according to the resource allocation plan and data quality tags to obtain a data cleaning report to implement data storage control.

[0015] The present invention obtains the original data stream and application demand information, and performs dynamic correlation analysis to construct a dynamically updated data association map, and finally generates a data feature vector. The method can comprehensively characterize the basic attributes, content information, usage and association relationship of the data with other data, lay a solid foundation for subsequent intelligent data management and resource optimization, and achieve more accurate data portraits and more effective feature extraction. Through intelligent quality evaluation, timeliness evaluation, access heat evaluation and application scenario weight calculation, the method can comprehensively consider multiple factors such as data quality, time value, use value and application scenario importance, and conduct a more comprehensive and accurate value evaluation of the data, and finally obtain data value scoring and data value level data, which provide a reliable basis for subsequent resource allocation and data cleaning, and achieve more intelligent data value evaluation and more flexible classification strategy. According to the data value level data, adaptive resource allocation can be performed, and limited storage resources can be preferentially allocated to high-value data, thereby improving resource utilization efficiency and ensuring the access performance of key data. Through resource demand prediction and dynamic resource pool management, it can better adapt to changes in data environment and application requirements, and realize elastic scaling and optimal configuration of resources. Intelligent data cleaning based on resource allocation schemes and data quality labels can safely and effectively clean low-value and low-quality data, free up storage space, improve storage efficiency, and reduce storage costs. At the same time, generating a cleaning report can help users understand the cleaning effect and provide a reference for subsequent strategy optimization. Therefore, the present invention provides a storage control method based on big data, which realizes more intelligent, dynamic and efficient big data storage control through the key technologies of multidimensional data profiling, intelligent quality assessment, dynamic value grading and adaptive resource allocation, solves the limitations of traditional methods in static data management and resource allocation, improves storage resource utilization, data access performance and overall system efficiency, and better meets the data management needs in the big data era.

[0016] Preferably, step S1 comprises the following steps:

[0017] Step S11: obtaining the original data stream and application requirement information; extracting metadata from the original data stream to obtain metadata;

[0018] Step S12: extracting content features from the original data stream according to the metadata to obtain content features;

[0019] Step S13: Analyze the access pattern according to the metadata to obtain the access pattern;

[0020] Step S14: Perform data dynamic correlation analysis on metadata and content features according to access patterns and application demand information to obtain a dynamically updated data correlation graph;

[0021] Step S15: Generate feature vectors for metadata, content features, access patterns, and data association graphs to obtain data feature vectors.

[0022] The present invention can provide basic data for subsequent data analysis and processing by acquiring the original data stream and application demand information in real time and extracting metadata from the original data stream, so that the system can understand the basic attributes of the data, such as data type, size, format, etc., thereby laying the foundation for subsequent correlation analysis, quality assessment and value grading. According to the extracted metadata, different feature extraction algorithms are used for different types of data, which can more effectively capture the intrinsic information of the data. For example, statistical features are extracted for numerical data, keywords are extracted for text data, and deep learning features are extracted for image data, so that the data representation is richer and more comprehensive, which helps to more accurately evaluate the quality and value of data. By analyzing the access mode of data, such as access frequency, access time interval and access user group, the popularity and usage of data can be understood. This helps to distinguish frequently accessed hot data from infrequently accessed cold data, and provides an important basis for subsequent dynamic value grading and adaptive resource allocation, thereby optimizing storage resource utilization and data access performance. By combining access mode and application demand information, dynamic correlation analysis is performed on metadata and content features, and a dynamically updated data association map is constructed, the association relationship between data can be captured more accurately. For example, there may be associations between data that are frequently accessed together, and the associations between certain data are stronger in specific application scenarios. Dynamically updated graphs can reflect the ever-changing relationships between data, thereby improving the accuracy of data value assessment and the efficiency of resource allocation. Integrating metadata, content features, access patterns, and data association graphs into a unified data feature vector can more comprehensively describe all aspects of the data, including the basic attributes of the data, content information, usage, and associations with other data. This provides richer input information for subsequent intelligent quality assessment, dynamic value grading, and adaptive resource allocation, thereby improving the intelligence and efficiency of the entire storage control system.

[0023] Preferably, step S14 comprises the following steps:

[0024] Step S141: performing data preprocessing on metadata and content features to obtain preprocessed data;

[0025] Step S142: performing reference relationship identification on the pre-processed data to obtain a reference relationship set;

[0026] Step S143: Calculate data similarity on the preprocessed data to obtain a similarity matrix;

[0027] Step S144: performing time series relationship identification on the pre-processed data to obtain a time series relationship set;

[0028] Step S145: constructing a data association graph according to the reference relationship set, the similarity matrix and the temporal relationship set to obtain a data association graph;

[0029] Step S146: Perform dynamic association analysis on the data association graph according to the access mode and application requirement information to obtain a dynamically updated data association graph.

[0030] The present invention can eliminate the differences in different feature types and value ranges by preprocessing metadata and content features, such as normalization, digitization and feature vector splicing, and improve the accuracy and efficiency of subsequent correlation analysis. This allows different types of data to be compared and analyzed under a unified framework, so as to more effectively discover the correlation between data. By identifying the reference relationship between data, such as file paths, URL links and database foreign keys, a direct association between data can be established. This helps to understand the dependencies and hierarchical structures of data, for example, which pictures or videos are referenced by a web page, and which other records are referenced by a database record, so as to more completely characterize the correlation between data. By calculating the data similarity matrix, the similarity between data can be quantified. This helps to find data with similar content, such as similar pictures, similar texts or similar numerical data, even if there is no direct reference relationship between them, the potential association between them can be found through similarity analysis. By identifying the temporal relationship between data, such as the order of creation time and modification time, the time evolution law of data can be understood. This helps to analyze the life cycle and change trend of data, for example, which data is newly generated and which data has been modified, so as to better understand the dynamic change process of data. Integrating reference relationships, similarities, and temporal relationships into the data association graph can more comprehensively represent the associations between data. The graph structure can clearly show the various relationships between data and support the use of graph algorithms for deeper association analysis, such as discovering community structures, key nodes, and paths in the data. By dynamically adjusting the data association graph based on access patterns and application demand information, the graph can more accurately reflect the actual degree of association between data. For example, data that is frequently accessed together is more closely associated, and the associations between certain data in specific application scenarios are more important. Dynamically updated graphs can adapt to data changes and user needs, thereby improving the efficiency of data management and resource allocation.

[0031] Preferably, step S2 comprises the following steps:

[0032] Step S21: performing feature vector preprocessing on the data feature vector to obtain a preprocessed feature vector; performing quality assessment on the preprocessed feature vector using a preset machine learning model to obtain a quality score;

[0033] Step S22: Generate a data quality label according to the quality score to obtain a data quality label;

[0034] Step S23: performing timeliness evaluation and access popularity evaluation based on the data feature vector to obtain a timeliness score and an access popularity score;

[0035] Step S24: Obtain system resource load information; calculate the application scenario weight according to the data feature vector and the system resource load information to obtain the application scenario weight;

[0036] Step S25: quantify the value according to the timeliness score, access popularity score, application scenario weight and data quality label to obtain a preliminary value score; standardize the preliminary value score to obtain a data value score;

[0037] Step S26: Formulate a grading strategy based on the data value score, system resource load information, and application demand information to obtain a dynamic grading strategy;

[0038] Step S27: Classify the data according to the dynamic classification strategy and data value score to obtain data value level data.

[0039] The present invention can improve the evaluation accuracy and efficiency of the machine learning model by preprocessing the data feature vector, such as missing value filling and standardization. By using a preset machine learning model to evaluate the quality of the preprocessed feature vector, low-quality data, such as duplicate data, incomplete data, etc., can be automatically identified, thereby providing a basis for subsequent data cleaning and value grading. Generating data quality labels (such as high, medium, and low) according to the quality score can simplify the subsequent data processing process. The data quality label can be used as an important reference factor for data value evaluation and used to guide the formulation of data cleaning strategies, such as giving priority to cleaning low-quality data. Through timeliness evaluation and access heat evaluation, the time value and use value of data can be quantified. The timeliness score reflects the timeliness of the data, for example, news data will reduce its value over time. The access heat score reflects the popularity of the data, and frequently accessed data usually has a higher value. Both scores are important components of data value evaluation. Obtaining system resource load information, such as CPU utilization, memory usage, and storage space occupancy, can enable the system to dynamically adjust the data management strategy according to the current resource status. Combining the data feature vector and the system resource load, the application scenario weight can be calculated. For example, when resources are tight, the weight of high-priority application scenarios can be increased to prioritize the performance of key applications. Taking into account the timeliness score, access popularity score, application scenario weight, and data quality label, the value of data can be more comprehensively evaluated. Through value quantification and standardization, the data value of different dimensions can be converted into a unified score, which is convenient for subsequent data classification and resource allocation. According to the data value score, system resource load, and application demand information, a dynamic classification strategy can be formulated, and the storage level of data can be flexibly adjusted according to the data value and system resource status. For example, when system resources are tight, some high-value data can be migrated from the high-speed storage layer to the medium-speed storage layer to balance performance and cost. Data can be classified according to the dynamic classification strategy and data value score, so that data can be stored in the most appropriate storage layer, such as storing high-value data in the high-speed storage layer and low-value data in the low-speed storage layer. This can maximize the utilization of storage resources and meet the performance requirements of different application scenarios.

[0040] Preferably, step S23 includes the following steps:

[0041] Step S231: acquiring time features of the data feature vector to obtain time features; calculating time differences of the time features to obtain time difference values;

[0042] Step S232: Calculate the initial timeliness score using the time difference and the preset time decay function to obtain the initial timeliness score; perform historical access frequency statistics on the data feature vector to obtain the historical access frequency;

[0043] Step S233: adjusting the decay function parameters according to the historical access frequency and the initial timeliness score to obtain an adjusted decay function; calculating the final timeliness score using the adjusted decay function and the time difference to obtain a timeliness score;

[0044] Step S234: acquiring access features of the data feature vector to obtain access features; calculating an initial heat score using the access features and a preset heat calculation method to obtain an initial heat score;

[0045] Step S235: acquiring associated data from the data feature vector to obtain an associated data list; calculating associated data value according to the associated data list to obtain an associated data value list;

[0046] Step S236: performing an associated data impact calculation based on the associated data value list and the associated data list to obtain an associated data impact value; performing a final heat score calculation on the associated data impact value and the initial heat score to obtain an access heat score.

[0047] The present invention can provide basic data for subsequent timeliness evaluation by obtaining the time characteristics of data (such as creation timestamp and last modification timestamp) and calculating the time difference. The time difference is an important indicator to measure the freshness of data. The larger the time difference, the more outdated the data is. The initial timeliness score is calculated using the time difference and the preset time decay function, and the aging degree of the data can be quantified. At the same time, the historical access frequency of the statistical data can understand the historical use of the data and provide a basis for the subsequent adjustment of the decay function parameters. Adjusting the decay function parameters according to the historical access frequency can make the timeliness evaluation more flexible and accurate. For example, for data with a high access frequency, even if the time is long, its value may be high, so its timeliness decay speed can be slowed down. By obtaining the access characteristics of the data, such as the number of visits in the recent period, and using the preset heat calculation method (such as the moving average method), the initial heat score of the data can be calculated, thereby preliminarily evaluating the popularity of the data. By obtaining the list of associated data of the data and calculating the value of the associated data, the influence of the association relationship between the data on the value of the data can be considered. For example, a data itself has a low access frequency, but if it is associated with many high-value data, its own value may also be high. Combining the impact value of the associated data with the initial heat score can more comprehensively evaluate the access heat of the data. This can avoid the one-sidedness of evaluating data heat based only on access frequency, and more accurately reflect the actual value of the data.

[0048] Preferably, step S24 comprises the following steps:

[0049] Step S241: extracting scene-related features from the data feature vector to obtain scene-related features;

[0050] Step S242: Identify the application scenario according to the scenario-related features to obtain an application scenario identifier;

[0051] Step S243: performing initial scene weight analysis according to the application scene identifier using a preset scene weight table to obtain an initial scene weight;

[0052] Step S244: obtaining user priority setting information; dynamically adjusting the initial scene weight according to the system resource load information and the user priority setting information to obtain the application scene weight.

[0053] The present invention can provide necessary feature information for subsequent application scenario identification by extracting scenario-related features, such as data types, sources, tags, and access modes. By analyzing these features, the system can better understand the purpose and use environment of the data. According to the extracted scenario-related features, the application scenarios of the data are identified and the corresponding application scenario identifiers are assigned, and the data can be divided into different application categories. This helps to distinguish the value and priority of data in different application scenarios. For example, data analyzed in real time usually has a higher priority than historical archived data. Using a preset scenario weight table, initial weights can be assigned to different application scenarios to reflect the relative importance of different application scenarios. This provides a basis for the subsequent dynamic adjustment of scenario weights and ensures that the system can still perform data value assessment according to predefined rules in the absence of other information. In combination with user priority setting information and system resource load information, the application scenario weights are dynamically adjusted, so that the system can adapt to different user needs and system states more flexibly. For example, in the case of tight system resources, the weights of low-priority application scenarios can be reduced, and resources can be preferentially allocated to high-priority application scenarios, thereby ensuring the performance of key applications.

[0054] Preferably, step S26 comprises the following steps:

[0055] Step S261: Determine the classification threshold according to the data value score, system resource load information and application demand information to obtain the classification threshold;

[0056] Step S262: selecting a data storage medium according to the classification threshold and the data value score, and obtaining a storage medium matching table;

[0057] Step S263: generating an initial classification strategy according to the classification threshold and the storage medium matching table to obtain an initial classification strategy;

[0058] Step S264: dynamically optimize the initial grading strategy according to the storage resource load information and the application demand information to obtain a dynamic grading strategy.

[0059] The present invention determines the classification threshold by comprehensively considering the data value score, system resource load and application demand information, so as to make the classification strategy more flexible and adaptive. For example, when the system resource load is high, the classification threshold can be increased to store more data in the storage medium of the lower level, thereby alleviating the system pressure; conversely, when the system resources are sufficient, the classification threshold can be lowered to store more data in the storage medium of higher performance to improve the data access performance. At the same time, considering the application demand information, such as the requirements of some applications for data reliability or access delay, the setting of the classification threshold can be further refined to better meet the needs of different applications. According to the classification threshold and the data value score, a suitable storage medium is selected for each data, and a storage medium matching table is generated, so as to ensure that the data is stored in a storage level that matches its value. High-value data will be stored in a storage medium with higher performance and higher reliability, while low-value data will be stored in a storage medium with lower cost, so as to achieve a balance between performance and cost. According to the classification threshold and the storage medium matching table, the initial classification strategy is generated, and the data value can be associated with a specific storage device and storage pool. This provides a basis for subsequent dynamic optimization and ensures that data can still be graded according to predefined policies when the system is started or there is no dynamic adjustment. Dynamic optimization of the initial grading strategy based on storage resource load information and application demand information can make the grading strategy more suitable for the actual operating environment, improve resource utilization efficiency and meet the performance requirements of different applications. For example, when a storage device is overloaded, the grading strategy can be dynamically adjusted to migrate part of the data to other devices with lower loads to avoid performance bottlenecks; when an application has more stringent requirements on data access latency, the data of the application can be stored preferentially on storage devices with higher performance.

[0060] Preferably, step S3 comprises the following steps:

[0061] Step S31: performing resource demand prediction on the data value level data to obtain a resource demand prediction result;

[0062] Step S32: performing resource pool management according to the resource demand prediction result to obtain resource pool configuration data;

[0063] Step S33: Formulate a resource allocation strategy based on the resource demand prediction result and the resource pool configuration data to obtain a resource allocation strategy;

[0064] Step S34: Allocate resources according to the resource allocation strategy and the data value level data to obtain a resource allocation plan.

[0065] The present invention can estimate the storage space, bandwidth and IOPS required for different storage levels in the future by predicting the resource demand of data value level data, thereby providing forward-looking guidance for resource pool management and resource allocation strategy formulation, avoiding the occurrence of insufficient or excessive resources, and improving resource utilization efficiency. According to the resource demand prediction results, the resource pool management can dynamically adjust the capacity, bandwidth and IOPS limit of the resource pool to ensure that the resource pool can meet the predicted resource demand. For example, if it is predicted that the resource demand of a certain storage level will increase, the capacity of the resource pool of this level can be expanded in advance, or resources can be allocated from other levels of resource pools to avoid the emergence of resource bottlenecks. According to the resource demand prediction results and the resource pool configuration data, the resource allocation strategy is formulated, and the data of different value levels can be reasonably allocated to different resource pools, ensuring that high-value data can obtain sufficient resource guarantees, while avoiding low-value data from occupying too many resources. This prediction-based resource allocation strategy can improve resource utilization efficiency and meet the performance requirements of data of different value levels. According to the resource allocation strategy and data value level data, each data can be accurately allocated to the corresponding resource pool, and the corresponding storage space, bandwidth and IOPS are allocated to it. This enables refined management of resources, avoids resource waste, and ensures the performance requirements of data at different value levels.

[0066] Preferably, step S4 comprises the following steps:

[0067] Step S41: Formulate a cleanup strategy based on the resource allocation plan and the data quality label to obtain a data cleanup strategy;

[0068] Step S42: Identify data according to the data cleaning strategy and data value level to obtain a list of data to be cleaned;

[0069] Step S43: performing data cleaning on the data list to be cleaned according to the data cleaning strategy to obtain a cleaning result;

[0070] Step S44: Generate a cleanup report based on the cleanup results to obtain a data cleanup report.

[0071] The present invention can clean up data more specifically by formulating a cleaning strategy according to a resource allocation scheme and a data quality label, for example, data stored in a low-speed storage layer and of low quality is cleaned up first. This cleaning strategy based on resource allocation and data quality can effectively improve the utilization of storage space and avoid the accidental deletion of important data. By identifying the data to be cleaned according to the cleaning strategy and the data value level, it can be ensured that only qualified data, such as data of low value and poor quality, is cleaned up. In combination with the data value level, it is possible to avoid the accidental cleaning of high-value data, even if the data is of low quality or occupies a large amount of storage space. Performing cleaning operations such as deleting or compressing the identified data to be cleaned according to the cleaning strategy can effectively release storage space and improve storage efficiency. According to different data types and cleaning strategies, different cleaning methods can be adopted, such as deleting duplicate data, compressing obsolete data, etc., so as to minimize the storage space occupied. A cleaning report is generated, which can record detailed information of the cleaning process, such as the amount of data cleaned, the storage space saved, the cleaning time, etc., so as to facilitate users to understand the cleaning effect and the system operation status. The cleaning report is also helpful for subsequent analysis and optimization of the cleaning strategy, for example, adjusting the parameters or rules of the cleaning strategy according to the problems reflected in the cleaning report, thereby further improving the cleaning efficiency.

[0072] Preferably, the present invention further provides a storage control device based on big data, comprising an intelligent SSD controller, a storage level sensor, an SCM memory, a hierarchical storage medium, a system resource management module and a user interface, wherein the storage level sensor and the SCM memory are integrated inside the SSD controller, the hierarchical storage medium is connected to the intelligent SSD controller via a storage interface, the system resource management module is connected to the intelligent SSD controller via a system summary, and the user interface is connected to the system resource management module via a network interface, and is used to execute the storage control method based on big data as described above, wherein the intelligent SSD controller of the storage control device based on big data comprises:

[0073] The data dynamic association analysis module is used to obtain the original data stream and application requirement information; perform data dynamic association analysis on the original data stream according to the application requirement information to obtain a dynamically updated data association map; generate a feature vector for the dynamically updated data association map to obtain a data feature vector;

[0074] The dynamic value grading module is used to perform intelligent quality assessment on data feature vectors to obtain data quality labels; perform timeliness assessment based on data feature vectors and access heat assessment to obtain timeliness scores and access heat scores; obtain system resource load information; calculate application scenario weights based on data feature vectors and system resource load information to obtain application scenario weights; perform data value assessment based on timeliness scores, access heat scores, application scenario weights, and data quality labels to obtain data value scores; perform data strategy grading processing based on data value scores to obtain data value grade data;

[0075] An adaptive resource allocation module is used to perform adaptive resource allocation according to data value level data to obtain a resource allocation plan;

[0076] The intelligent data cleaning module is used to perform intelligent data cleaning according to the resource allocation plan and data quality labels to obtain a data cleaning report. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a module block diagram of a storage control device based on big data;

[0078] Figure 2 A schematic diagram of the steps of a storage control method based on big data;

[0079] Figure 3 Detailed implementation flow chart of step S2.

[0080] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0081] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0082] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0083] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0084] To achieve this, please refer to Figures 1 to 3 , a storage control method based on big data, comprising the following steps:

[0085] Step S1: obtaining the original data stream and application requirement information; performing data dynamic correlation analysis on the original data stream according to the application requirement information to obtain a dynamically updated data correlation map; generating a feature vector for the dynamically updated data correlation map to obtain a data feature vector;

[0086] Step S2: Perform intelligent quality assessment on the data feature vector to obtain a data quality label; perform timeliness assessment based on the data feature vector and perform access heat assessment to obtain a timeliness score and an access heat score; obtain system resource load information; perform application scenario weight calculation based on the data feature vector and the system resource load information to obtain an application scenario weight; perform data value assessment based on the timeliness score, access heat score, application scenario weight and data quality label to obtain a data value score; perform data strategy classification processing based on the data value score to obtain data value level data;

[0087] Step S3: adaptively allocate resources according to the data value level data to obtain a resource allocation plan;

[0088] Step S4: Perform intelligent data cleaning according to the resource allocation plan and data quality tags to obtain a data cleaning report to implement data storage control.

[0089] In the embodiment of the present invention, reference Figure 1 FIG. 1 is a schematic diagram of a step flow chart of a storage control method based on big data of the present invention. In this example, the storage control method based on big data includes the following steps:

[0090] Step S1: obtaining the original data stream and application requirement information; performing data dynamic correlation analysis on the original data stream according to the application requirement information to obtain a dynamically updated data correlation map; generating a feature vector for the dynamically updated data correlation map to obtain a data feature vector;

[0091] In an embodiment of the present invention, the storage-level sensor in the intelligent SSD controller captures the original data stream, and the system resource management module receives the application requirement information (such as data type, access mode, retention policy, etc.) provided by the user. The data dynamic association analysis module first extracts metadata and content features, and then analyzes the access mode. Next, based on the application requirement information, the metadata and content features are dynamically analyzed for relevance, and a dynamically updated data association graph is constructed. The graph is stored in a graph database, where nodes represent data and edges represent relationships between data (references, similarities, and temporal relationships). The weight of the edge is determined by the access mode and application requirement information. Finally, the graph embedding vectors of metadata, content features, access modes, and data association graphs are integrated to generate a data feature vector, which is then standardized and stored in the SCM memory.

[0092] Step S2: Perform intelligent quality assessment on the data feature vector to obtain a data quality label; perform timeliness assessment based on the data feature vector and perform access heat assessment to obtain a timeliness score and an access heat score; obtain system resource load information; perform application scenario weight calculation based on the data feature vector and the system resource load information to obtain an application scenario weight; perform data value assessment based on the timeliness score, access heat score, application scenario weight and data quality label to obtain a data value score; perform data strategy classification processing based on the data value score to obtain data value level data;

[0093] In an embodiment of the present invention, after the dynamic value grading module preprocesses the data feature vector (filling missing values, Z-score standardization), it uses the pre-trained SVM model to perform quality assessment to obtain a quality score, and generates a data quality label (high, medium, low) based on the score. At the same time, the module calculates the time difference based on the data creation time and the current time, uses the exponential decay function to calculate the initial timeliness score, and adjusts the decay coefficient based on the historical access frequency to finally calculate the timeliness score. The access heat score is obtained by calculating the moving average of the number of visits in the last 7 days to obtain the initial heat score, and is adjusted in combination with the value score of the associated data to obtain the final score. After obtaining the system resource load information, the application scenario weight is dynamically adjusted based on the application scenario identifier extracted from the data feature vector and the preset scenario weight table, combined with the user priority setting and the system resource load. Finally, the timeliness score, access heat score, application scenario weight and data quality label score are combined to calculate the preliminary value score, and the final data value score is obtained after Min-Max standardization. Based on the data value score, system resource load information and application demand information, the classification threshold and storage media matching table are determined, and the initial classification strategy is generated. Dynamic optimization is performed based on real-time resource load information and application requirements, and finally a dynamic classification strategy is obtained. Data is classified according to the strategy and data value score to obtain data value level data.

[0094] Step S3: adaptively allocate resources according to the data value level data to obtain a resource allocation plan;

[0095] In an embodiment of the present invention, the adaptive resource allocation module uses a time series analysis method to predict the resource demand (data volume growth, bandwidth, IOPS) of each storage layer in the future based on the data value level data. Then, the resource pool configuration is dynamically adjusted according to the prediction results, for example, resources are allocated from other layers when resources are insufficient. Next, a resource allocation strategy is formulated based on the resource demand prediction results and the resource pool configuration data to specify how data of different value levels are allocated to different resource pools. Finally, based on the resource allocation strategy and data value level data, each data is allocated to the corresponding resource pool, and the allocated resource information (resource pool ID, storage space, bandwidth, IOPS) is recorded to generate a resource allocation plan.

[0096] Step S4: Perform intelligent data cleaning according to the resource allocation plan and data quality labels to obtain a data cleaning report to achieve data storage control;

[0097] In an embodiment of the present invention, the intelligent data cleaning module formulates a data cleaning strategy based on the resource allocation scheme (storage level, resource pool information) and data quality labels, for example, deciding which data to delete or compress based on the storage space occupancy rate and data quality level. Then, according to the cleaning strategy and data value level, the data that meets the cleaning conditions is identified, and a list of data to be cleaned is generated. Next, a cleaning operation (deletion or compression) is performed on the data in the list, and the cleaning results are recorded. Finally, a cleaning report is generated based on the cleaning results, including statistical information such as the amount of data cleaned, the storage space saved, and the cleaning time.

[0098] Preferably, step S1 comprises the following steps:

[0099] Step S11: obtaining the original data stream and application requirement information; extracting metadata from the original data stream to obtain metadata;

[0100] Step S12: extracting content features from the original data stream according to the metadata to obtain content features;

[0101] Step S13: Analyze the access pattern according to the metadata to obtain the access pattern;

[0102] Step S14: Perform data dynamic correlation analysis on metadata and content features according to access patterns and application demand information to obtain a dynamically updated data correlation graph;

[0103] Step S15: Generate feature vectors for metadata, content features, access patterns, and data association graphs to obtain data feature vectors.

[0104] In an embodiment of the present invention, the storage-level sensor in the intelligent SSD controller captures the raw data stream from the layered storage medium (e.g., 3D XPoint, SSD, HDD) in real time. At the same time, the system resource management module receives user-defined application requirement information, such as data type, expected access mode, data retention policy, etc. through a user interface. The application requirement information is stored in JSON format. The storage-level sensor uses a built-in parser to extract metadata from the raw data stream. For structured data (such as database records), the extracted metadata includes field name, data type, data length, etc. For unstructured data (such as image files), the extracted metadata includes file format, resolution, creation time, etc. The extracted metadata is stored in the SCM memory and organized in the form of key-value pairs, where the key is the data unique identifier and the value is the corresponding metadata information.

[0105] The data dynamic association analysis module in the intelligent SSD controller selects the corresponding content feature extraction algorithm based on the metadata. For numerical data, its statistical features such as mean, variance, maximum, and minimum are calculated. For text data, natural language processing technology (NLP) is used for word segmentation, stem extraction, and stop word filtering, and then the TF-IDF algorithm is used to calculate the keyword weight, and the keywords and their weights are used as text content features. For image data, a dedicated AI chip integrated in the intelligent SSD controller is used to run a pre-trained convolutional neural network (CNN) model to extract the deep learning feature vector of the image. The extracted content features are stored in the SCM memory together with the corresponding metadata.

[0106] The access pattern analysis module built into the intelligent SSD controller uses metadata to analyze the access pattern of data. This module records the access timestamp and access user ID of each data block. By analyzing the data access records over a period of time, the access frequency, access interval time, and access user group of each data block are counted. The access frequency is calculated using the sliding window method, and the window size is set to the last 7 days. The access interval time is calculated as the time difference between two consecutive accesses. The access user group counts the number of different user IDs that have accessed the data block. Finally, the access frequency, access interval time, and access user group are stored in the SCM memory as the access pattern of the data.

[0107] The data dynamic association analysis module performs dynamic association analysis on metadata and content features based on access patterns and application demand information. First, each data block is represented as a node in the graph database, and the node attributes include metadata and content features. Then, edges are established between nodes based on the reference relationship, similarity, and timing relationship between data. Reference relationships are identified by analyzing pointers or links in the data content. Similarity is determined by calculating the cosine similarity between data content feature vectors, with a threshold set to 0.8. The timing relationship is determined based on the data creation time and modification time. Access patterns and application demand information are used to adjust the edge weights, such as higher edge weights between data that are frequently accessed together, and higher edge weights between data that meet specific application requirements. Finally, a dynamically updated data association graph is constructed and stored in the graph database within the intelligent SSD controller.

[0108] The data dynamic association analysis module integrates metadata, content features, access patterns, and dynamically updated data association graphs to generate data feature vectors. Specifically, metadata, content features, and access patterns are converted into numerical vectors respectively. For the data association graph, a graph embedding algorithm (such as Node2Vec) is used to represent each data node as a low-dimensional vector. These vectors are then concatenated together to form the final data feature vector. In order to avoid the impact of differences in the value ranges of features of different dimensions on subsequent processing, the feature vectors are standardized, for example, the Min-Max normalization method is used to scale the value range of each feature to between [0,1]. The final generated data feature vector is stored in the SCM memory.

[0109] Preferably, step S14 comprises the following steps:

[0110] Step S141: performing data preprocessing on metadata and content features to obtain preprocessed data;

[0111] Step S142: performing reference relationship identification on the pre-processed data to obtain a reference relationship set;

[0112] Step S143: Calculate data similarity on the preprocessed data to obtain a similarity matrix;

[0113] Step S144: performing time series relationship identification on the pre-processed data to obtain a time series relationship set;

[0114] Step S145: constructing a data association graph according to the reference relationship set, the similarity matrix and the temporal relationship set to obtain a data association graph;

[0115] Step S146: Perform dynamic association analysis on the data association graph according to the access mode and application requirement information to obtain a dynamically updated data association graph.

[0116] In an embodiment of the present invention, the data dynamic association analysis module receives metadata and content features. Metadata is stored in the form of key-value pairs, and content features are stored in the form of vectors. The preprocessing process first normalizes the numerical metadata, for example, using Min-Max scaling to scale all numerical features to the [0,1] interval. Then, the text-type metadata and content features are converted into numerical vectors, for example, using a word embedding model (Word2Vec) to convert text into vectors. Finally, all feature vectors are spliced ​​into a unified feature vector as preprocessed data. The preprocessed data is stored in the SCM memory for use in subsequent steps.

[0117] The data dynamic association analysis module analyzes specific fields in the preprocessed data, such as file paths, URL links, database foreign keys, etc., to identify the reference relationship between data. For structured data, the foreign key relationship is identified by parsing the database table structure and added to the reference relationship collection. For unstructured data, such as HTML files, the ` The ` tag extracts the URL link and adds the link relationship to the reference relationship set. The reference relationship set is stored in the form of a triple of (data ID1, data ID2, relationship type), indicating that data ID1 references data ID2 and the relationship type is "reference".

[0118] The data dynamic association analysis module uses the cosine similarity algorithm to calculate the similarity between the feature vectors of the preprocessed data. All data feature vectors are combined into a matrix, and then the cosine similarity matrix of the matrix is ​​calculated. The similarity matrix is ​​a symmetric matrix, in which each element represents the similarity score between two data, and the value range is [0,1]. The similarity threshold is set to 0.85. If the similarity score between two data is higher than this threshold, they are considered similar.

[0119] The data dynamic association analysis module extracts timestamp information such as data creation time and last modification time from metadata. Based on the timestamp information, the temporal relationship between data is determined. For example, if the creation time of data A is later than the creation time of data B, data A is considered to be created after data B, and this temporal relationship is added to the temporal relationship set. The temporal relationship set is stored in the form of a triple of (data ID1, data ID2, relationship type), and the relationship type is "first and last", indicating that data ID1 is created after data ID2.

[0120] The data dynamic association analysis module uses graph database technology to build a data association graph. Each data is represented as a node in the graph database, and the node attributes include preprocessed data. According to the reference relationship set, similarity matrix and time series relationship set, edges are established between nodes. For example, if data A references data B, a directed edge is established between nodes A and B, and the edge type is "reference". If the similarity between data A and data B is higher than the threshold, an undirected edge is established between nodes A and B, and the edge type is "similar". If data A is created after data B, a directed edge is established between nodes A and B, and the edge type is "previous".

[0121] The data dynamic association analysis module performs dynamic association analysis on the data association graph based on the access pattern and application demand information. For example, if two data are often accessed by the same user in a short period of time, the weight of the edge between them is increased. If the application demand information specifies that certain types of data have a higher association, the weight of the edge between these data is adjusted accordingly. In this way, a dynamically updated data association graph is constructed to reflect the ever-changing association relationship between data. The dynamically updated data association graph is continuously stored in the graph database.

[0122] Preferably, step S2 comprises the following steps:

[0123] Step S21: performing feature vector preprocessing on the data feature vector to obtain a preprocessed feature vector; performing quality assessment on the preprocessed feature vector using a preset machine learning model to obtain a quality score;

[0124] Step S22: Generate a data quality label according to the quality score to obtain a data quality label;

[0125] Step S23: performing timeliness evaluation and access popularity evaluation based on the data feature vector to obtain a timeliness score and an access popularity score;

[0126] Step S24: Obtain system resource load information; calculate the application scenario weight according to the data feature vector and the system resource load information to obtain the application scenario weight;

[0127] Step S25: quantify the value according to the timeliness score, access popularity score, application scenario weight and data quality label to obtain a preliminary value score; standardize the preliminary value score to obtain a data value score;

[0128] Step S26: Formulate a grading strategy based on the data value score, system resource load information, and application demand information to obtain a dynamic grading strategy;

[0129] Step S27: Classify the data according to the dynamic classification strategy and data value score to obtain data value level data.

[0130] As an example of the present invention, refer to Figure 3 As shown, in this example, step S2 includes:

[0131] Step S21: performing feature vector preprocessing on the data feature vector to obtain a preprocessed feature vector; performing quality assessment on the preprocessed feature vector using a preset machine learning model to obtain a quality score;

[0132] In an embodiment of the present invention, a dynamic value grading module receives feature vectors from data. The preprocessing process first fills missing values ​​in the feature vector, and uses the feature mean to fill missing values. Then, the feature vector is standardized using the Z-score standardization method, and the value of each feature is converted into a distribution with a mean of 0 and a standard deviation of 1. The preprocessed feature vector is input into a preset machine learning model for quality assessment. The machine learning model is a pre-trained support vector machine (SVM) classification model, and the model is stored in an SCM memory. The model input is the preprocessed feature vector, and the output is a data quality score, and the score ranges from 0 to 1. The higher the value, the higher the data quality.

[0133] Step S22: Generate a data quality label according to the quality score to obtain a data quality label;

[0134] In the embodiment of the present invention, a data quality label is generated according to the data quality score. Three quality levels are set: high, medium, and low. If the quality score is greater than or equal to 0.8, the data quality label is "high"; if the quality score is between 0.5 and 0.8, the data quality label is "medium"; if the quality score is less than 0.5, the data quality label is "low". The data quality label is stored in the SCM memory together with the corresponding data feature vector.

[0135] Step S23: performing timeliness evaluation and access popularity evaluation based on the data feature vector to obtain a timeliness score and an access popularity score;

[0136] In the embodiment of the present invention, the data creation timestamp and the most recent access timestamp are extracted from the data feature vector. The timeliness evaluation uses an exponential decay function: `TimelinessScore=exp(-k*(current time-creation time))`, where k is the decay coefficient, set to 0.01. The access popularity evaluation uses the access frequency in the last 7 days: `AccessPopularity Score = number of visits in the last 7 days / 7`.

[0137] Step S24: Obtain system resource load information; calculate the application scenario weight according to the data feature vector and the system resource load information to obtain the application scenario weight;

[0138] In the embodiment of the present invention, the system resource load information is obtained from the system resource management module, including CPU utilization, memory usage, storage space occupancy, etc. The application scenario weight is calculated based on the application scenario identifier (e.g., "real-time analysis", "historical archiving") extracted from the data feature vector and combined with the system resource load information. For example, if the system storage space occupancy is high, the weight of the "historical archiving" application scenario is reduced, and the weight of the "real-time analysis" application scenario is increased to give priority to the storage resources of high-value data. The application scenario weight value ranges from 0 to 1.

[0139] Step S25: quantify the value according to the timeliness score, access popularity score, application scenario weight and data quality label to obtain a preliminary value score; standardize the preliminary value score to obtain a data value score;

[0140] In the embodiment of the present invention, the preliminary value score is calculated according to the following formula: `Preliminary Value Score = Timeliness Score * 0.3 + Access Heat Score * 0.5 + Application Scenario Weight * 0.2 + Data Quality Label Score`, where the data quality label score is: high is 1, medium is 0.5, and low is 0.1. Then, the preliminary value score is normalized to the [0,1] interval using the Min-Max normalization method to obtain the final data value score.

[0141] Step S26: Formulate a grading strategy based on the data value score, system resource load information, and application demand information to obtain a dynamic grading strategy;

[0142] In an embodiment of the present invention, a dynamic grading strategy is formulated based on the data value score, system resource load information, and application demand information (such as reliability requirements for data storage). Three storage tiers are set: a high-speed storage tier (3DXPoint), a medium-speed storage tier (SSD), and a low-speed storage tier (HDD). Based on the data value score and application demand information, the storage tier corresponding to data of different value levels is determined. For example, high-value data is stored in the high-speed storage tier, and low-value data is stored in the low-speed storage tier. If the system resource load is high, for example, the high-speed storage tier is insufficient, the threshold of the high-value data storage tier is appropriately lowered, and part of the high-value data is stored in the medium-speed storage tier.

[0143] Step S27: Classify the data according to the dynamic classification strategy and data value score to obtain data value level data.

[0144] In an embodiment of the present invention, data is graded according to a dynamic grading strategy and a data value score. The dynamic grading strategy defines the storage tiers corresponding to data in different value ranges. For example, assume that the dynamic grading strategy stipulates that data with a value score greater than or equal to 0.8 is stored in a high-speed storage layer (3D XPoint), data with a value score between 0.5 and 0.8 is stored in a medium-speed storage layer (SSD), and data with a value score less than 0.5 is stored in a low-speed storage layer (HDD). The dynamic value grading module compares the data value score of each data with the threshold in the dynamic grading strategy to determine the storage tier to which the data belongs, and records the data and its corresponding storage tier information (e.g., "high speed", "medium speed", "low speed") as data value level data, which is stored in the SCM memory for subsequent adaptive resource allocation.

[0145] Preferably, step S23 includes the following steps:

[0146] Step S231: acquiring time features of the data feature vector to obtain time features; calculating time differences of the time features to obtain time difference values;

[0147] Step S232: Calculate the initial timeliness score using the time difference and the preset time decay function to obtain the initial timeliness score; perform historical access frequency statistics on the data feature vector to obtain the historical access frequency;

[0148] Step S233: adjusting the decay function parameters according to the historical access frequency and the initial timeliness score to obtain an adjusted decay function; calculating the final timeliness score using the adjusted decay function and the time difference to obtain a timeliness score;

[0149] Step S234: acquiring access features of the data feature vector to obtain access features; calculating an initial heat score using the access features and a preset heat calculation method to obtain an initial heat score;

[0150] Step S235: acquiring associated data from the data feature vector to obtain an associated data list; calculating associated data value according to the associated data list to obtain an associated data value list;

[0151] Step S236: performing an associated data impact calculation based on the associated data value list and the associated data list to obtain an associated data impact value; performing a final heat score calculation on the associated data impact value and the initial heat score to obtain an access heat score.

[0152] In an embodiment of the present invention, time-related features, such as the creation timestamp and the last modification timestamp of the data, are extracted from the data feature vector. The time feature is expressed in Unix timestamp format, that is, the number of seconds since 00:00:00 Coordinated Universal Time (UTC) on January 1, 1970. The current timestamp is subtracted from the creation timestamp to obtain the time difference in seconds. For example, if the current timestamp is 1678886400 and the creation timestamp of the data is 1676092800, the time difference is 2793600 seconds.

[0153] The initial timeliness score is calculated using the preset exponential time decay function: `Initial TimelinessScore=exp(-k*time difference)`, where the decay coefficient k is initially set to 0.0001. For example, if the time difference is 2793600 seconds, the initial timeliness score is about 0.76. At the same time, the module extracts historical access frequency data from the data feature vector, which records the number of times the data has been accessed in the past period of time (for example, the past 30 days). For example, if a piece of data has been accessed 15 times in the past 30 days, its historical access frequency is 15.

[0154] Adjust the decay coefficient k according to the historical access frequency. If the historical access frequency is high, lower the k value so that the timeliness score decays more slowly. For example, set a threshold. If the historical access frequency is greater than or equal to 20, halve the k value, that is, k=0.00005. Then, recalculate the timeliness score using the adjusted decay coefficient k: `Timeliness Score=exp(-k*time difference)`. For example, using the adjusted k value of 0.00005 and the time difference of 2793600 seconds, the final timeliness score is calculated to be approximately 0.87.

[0155] Extract access features from the data feature vector, such as the number of visits in the last 7 days. Use a preset heat calculation method, such as the moving average method, to calculate the initial heat score. For example, if the number of visits in the last 7 days is [2, 3, 1, 4, 2, 5, 3], the initial heat score is calculated as (2+3+1+4+2+5+3) / 7=2.86.

[0156] Extract the information of the associated data from the data feature vector. For example, obtain the list of other data IDs directly connected to the current data through the data association graph to obtain the associated data list. Then, according to the calculated data value score corresponding to each data ID in the associated data list, obtain the associated data value list. For example, if the associated data list is [Data_ID_1, Data_ID_2, Data_ID_3], and the corresponding value scores are [0.9, 0.7, 0.6] respectively, then the associated data value list is [0.9, 0.7, 0.6].

[0157] Calculate the associated data influence value based on the associated data value list. For example, calculate the average value of the associated data value list as the associated data influence value: (0.9+0.7+0.6) / 3=0.73. Finally, take the weighted average of the associated data influence value and the initial popularity score to get the final access popularity score: `Access Popularity Score=Initial Popularity Score*0.7+Associated Data Influence Value*0.3`. For example, if the initial popularity score is 2.86 and the associated data influence value is 0.73, the final access popularity score is about 2.22.

[0158] Preferably, step S24 comprises the following steps:

[0159] Step S241: extracting scene-related features from the data feature vector to obtain scene-related features;

[0160] Step S242: Identify the application scenario according to the scenario-related features to obtain an application scenario identifier;

[0161] Step S243: performing initial scene weight analysis according to the application scene identifier using a preset scene weight table to obtain an initial scene weight;

[0162] Step S244: obtaining user priority setting information; dynamically adjusting the initial scene weight according to the system resource load information and the user priority setting information to obtain the application scene weight.

[0163] In an embodiment of the present invention, features related to the application scenario are extracted from the data feature vector. These features may include data type, data source, data tag, access mode, etc. For example, for a video data, its scene-related features may include video encoding format (such as H.264), video resolution (such as 1080p), video tags (such as "education", "entertainment"), and average access frequency (such as 100 visits per day). These features are stored in the form of key-value pairs, such as {"data_type": "video", "encoding": "H.264", "resolution": "1080p", "tags": ["education", "entertainment"], "access_frequency": 100}.

[0164] Use predefined rules or machine learning models to identify application scenarios based on scenario-related features. For example, if the data type is video, the resolution is 1080p, the tag contains "education", and the access frequency is high, set the application scenario identifier to "online education". If the data type is log file, the source is server, and the access frequency is low, set the application scenario identifier to "system log". The application scenario identifier is a string type, such as "online education", "system log", "financial transaction", etc.

[0165] Use the preset scenario weight table to determine the initial scenario weight according to the application scenario identifier. The scenario weight table is a set of key-value pairs, where the key is the application scenario identifier and the value is the corresponding weight. For example, the scenario weight table can be defined as {"online education": 0.9, "system log": 0.2, "financial transaction": 0.8}. This means that the initial weight of the "online education" application scenario is 0.9, the initial weight of the "system log" application scenario is 0.2, and the initial weight of the "financial transaction" application scenario is 0.8. The weight value range is 0 to 1, and the higher the value, the more important the application scenario is.

[0166] Obtain user priority setting information from the user interface. Users can set priorities for different application scenarios, such as setting the priority of the "online education" application scenario to "high" and the priority of the "system log" application scenario to "low". At the same time, the module obtains system resource load information from the system resource management module, such as CPU utilization, memory usage, and storage space occupancy. The initial scenario weights are dynamically adjusted according to the system resource load and user priority settings. For example, if the system storage space occupancy rate is high, and the priority of the "online education" application scenario is "high", while the priority of the "system log" application scenario is "low", the weight of the "system log" application scenario is reduced, and the saved weight is allocated to the "online education" application scenario to ensure the data storage performance of the high-priority application scenario. The adjusted application scenario weights remain in the range of 0 to 1.

[0167] Preferably, step S26 comprises the following steps:

[0168] Step S261: Determine the classification threshold according to the data value score, system resource load information and application demand information to obtain the classification threshold;

[0169] Step S262: selecting a data storage medium according to the classification threshold and the data value score, and obtaining a storage medium matching table;

[0170] Step S263: generating an initial classification strategy according to the classification threshold and the storage medium matching table to obtain an initial classification strategy;

[0171] Step S264: dynamically optimize the initial grading strategy according to the storage resource load information and the application demand information to obtain a dynamic grading strategy.

[0172] In an embodiment of the present invention, a data value score, system resource load information (such as CPU utilization, memory usage, remaining capacity of each storage layer), and application requirement information (such as data reliability requirements, access delay requirements) are received from a system resource management module. The module presets three grading thresholds, corresponding to three storage layers: high speed, medium speed, and low speed. The initial thresholds are set as: high-speed layer threshold 0.8, medium-speed layer threshold 0.5. If the system resource load information shows that the remaining capacity of the high-speed storage layer is less than 10%, the high-speed layer threshold is increased to 0.85 to reduce the amount of data entering the high-speed storage layer. If the application requirement information specifies that the data reliability requirement is high, the high-speed layer threshold is reduced to 0.75 to store more high-value data in a more reliable high-speed storage layer. The final grading threshold is stored in the SCM memory.

[0173] Select the appropriate storage medium for each data based on the grading threshold and data value score. The grading threshold divides the data value score into different intervals, and each interval corresponds to a storage tier. For example, assuming the grading threshold is 0.8 for high speed and 0.5 for medium speed, data with a value score greater than or equal to 0.8 corresponds to the high-speed storage tier (such as 3D XPoint), data with a value score between 0.5 and 0.8 corresponds to the medium-speed storage tier (such as SSD), and data with a value score less than 0.5 corresponds to the low-speed storage tier (such as HDD). The correspondence between the data value score interval and the storage medium is recorded in the storage medium matching table, for example `{[0.8,1.0]: "3D XPoint", [0.5,0.8) : "SSD", [0.0,0.5) : "HDD"}`.

[0174] Generate an initial grading strategy based on the grading threshold and storage media matching table. The initial grading strategy defines which specific storage device and storage pool the data in different value score intervals will be stored in. For example, assuming that there are multiple SSD devices in the system to form a medium-speed storage layer, the initial grading strategy will allocate the data in the value score interval [0.5,0.8) to different SSD devices and corresponding storage pools based on the performance indicators of the SSD devices (such as read and write speed, life) and load conditions. The initial grading strategy is expressed in the form of a rule list, such as `[{value_range: [0.8,1.0], storage_device: "3D XPoint_1", storage_pool: "pool_1"}, {value_range: [0.5,0.8), storage_device: "SSD_1", storage_pool: "pool_2"}, ...]`.

[0175] The initial tiering strategy is dynamically optimized based on the latest storage resource load information (such as the real-time capacity, read and write speed, and IO latency of each storage device) and application demand information (such as the performance requirements of different application scenarios). For example, if the load of a certain SSD device is too high, resulting in increased data access latency, the tiering strategy is dynamically adjusted to migrate part of the data from the SSD device to other SSD devices or storage pools with lower loads. If the application demand information specifies that the data of a certain application scenario requires lower access latency, the data of the application scenario is preferentially stored in a storage device or storage pool with higher performance. The optimization goal of the dynamic tiering strategy is to maximize storage resource utilization and meet the performance requirements of different application scenarios. The optimized dynamic tiering strategy is still represented in the form of a rule list and updated to the SCM storage.

[0176] Preferably, step S3 comprises the following steps:

[0177] Step S31: performing resource demand prediction on the data value level data to obtain a resource demand prediction result;

[0178] Step S32: performing resource pool management according to the resource demand prediction result to obtain resource pool configuration data;

[0179] Step S33: Formulate a resource allocation strategy based on the resource demand prediction result and the resource pool configuration data to obtain a resource allocation strategy;

[0180] Step S34: Allocate resources according to the resource allocation strategy and the data value level data to obtain a resource allocation plan.

[0181] In an embodiment of the present invention, the adaptive resource allocation module receives data value level data, which includes the storage level (e.g., high speed, medium speed, low speed) to which each data belongs. Based on historical data and real-time data, the module uses a time series analysis method (e.g., ARIMA model) to predict the data volume growth trend of each storage level in the future (e.g., the next 24 hours). At the same time, the module analyzes the historical access patterns (e.g., access frequency, amount of data read) of each storage level data, and predicts the bandwidth and IOPS (input / output operations per second) required for each storage level in the future. The prediction results are expressed in numerical form. For example, the data volume of the high-speed storage layer will increase by 1TB in the next 24 hours, the required bandwidth will be 10Gbps, and the required IOPS will be 10,000.

[0182] Storage resource pools are managed based on resource demand forecast results. The storage resources in the system are divided into multiple resource pools, each of which corresponds to a storage tier and has specific capacity, bandwidth, and IOPS limits. The module adjusts the configuration parameters of each resource pool based on the forecast results. For example, if the data volume of the high-speed storage layer is predicted to grow by 1TB in the next 24 hours, and the current remaining capacity of the high-speed storage pool is less than 1TB, the module will transfer a portion of resources (such as 1TB) from resource pools of other storage tiers (such as the medium-speed storage tier) to the high-speed storage pool to meet the forecasted demand. The resource pool configuration data contains information such as the ID of each resource pool, the storage tier to which it belongs, capacity, bandwidth, and IOPS limits.

[0183] Develop a resource allocation strategy based on the resource demand forecast results and resource pool configuration data. The resource allocation strategy defines how to allocate data of different value levels to different resource pools. For example, the strategy stipulates that data with a value score greater than or equal to 0.8 is stored in a high-speed resource pool, and data with a value score between 0.5 and 0.8 is stored in a medium-speed resource pool, and data with a value score less than 0.5 is stored in a low-speed resource pool. At the same time, the strategy will dynamically adjust the resource allocation of resource pools based on the resource demand forecast results. For example, if the IOPS demand of a high-speed resource pool is predicted to exceed its IOPS limit, the strategy will limit the amount of data allocated to the high-speed resource pool, or allocate some high-value data to the medium-speed resource pool to avoid performance bottlenecks.

[0184] Resources are allocated according to the resource allocation strategy and data value level data. The module allocates each data to the corresponding resource pool according to its value score and the storage tier to which it belongs. For example, a data with a value score of 0.9 belongs to the high-speed storage tier. According to the resource allocation strategy, it will be allocated to the high-speed resource pool. The resource allocation plan records the resource pool ID corresponding to each data, the allocated storage space size, bandwidth, IOPS and other information. The resource allocation plan is stored in the SCM storage and is used to guide the actual storage and access operations of the data.

[0185] Preferably, step S4 comprises the following steps:

[0186] Step S41: Formulate a cleanup strategy based on the resource allocation plan and the data quality label to obtain a data cleanup strategy;

[0187] Step S42: Identify data according to the data cleaning strategy and data value level to obtain a list of data to be cleaned;

[0188] Step S43: performing data cleaning on the data list to be cleaned according to the data cleaning strategy to obtain a cleaning result;

[0189] Step S44: Generate a cleanup report based on the cleanup results to obtain a data cleanup report.

[0190] In an embodiment of the present invention, an intelligent data cleaning module receives a resource allocation scheme and a data quality label. The resource allocation scheme includes the storage tier and resource pool information of each data, and the data quality label includes the quality level (high, medium, low) of each data. The cleaning strategy is formulated according to the storage resource load and the data quality level. For example, if the storage space occupancy rate of the low-speed storage layer (such as HDD) exceeds 90%, a cleaning strategy is formulated: delete the data with a quality label of "low" and stored in the low-speed storage layer; compress the data with a quality label of "medium" and stored in the low-speed storage layer, and the compression algorithm uses the Gzip compression algorithm; for data with a quality label of "high", even if it is stored in the low-speed storage layer, it is not cleaned. The cleaning strategy is stored in the form of a rule list, for example, `[{quality_label: "low", storage_tier: "low speed", action: "delete"}, {quality_label: "medium", storage_tier: "low speed", action: "compress", compression_algorithm: "Gzip"},...]`.

[0191] According to the data cleaning strategy and data value level data, the data to be cleaned is identified. The module traverses all data and matches the data's quality label, storage layer and value score with the rules in the cleaning strategy. For example, a data with a quality label of "low", stored in a low-speed storage layer, and a value score lower than 0.3 meets the rule in the cleaning strategy of "delete data with a quality label of 'low' and stored in a low-speed storage layer", so the data is added to the list of data to be cleaned. The list of data to be cleaned contains all data IDs that meet the cleaning strategy.

[0192] According to the data cleaning strategy, clean up the data in the list of data to be cleaned. For example, for a certain data in the list of data to be cleaned, if the operation specified by the cleaning strategy is "delete", the module will permanently delete the data from the storage system; if the operation specified by the cleaning strategy is "compress", the module will use the specified compression algorithm (such as Gzip) to compress the data and store the compressed data in the specified storage location. The cleaning result records the cleaning operation and results of each data to be cleaned, such as successful deletion, successful compression, compressed data size and other information.

[0193] The intelligent data cleaning module generates a data cleaning report based on the cleaning results. The cleaning report contains statistical information such as the total amount of data cleaned, the amount of data cleaned at each storage layer, the amount of storage space saved during the cleaning process, and the execution time of each cleaning operation. For example, the report can contain the following information: a total of 10TB of data was cleaned, of which 8TB was cleaned at the low-speed storage layer and 2TB was cleaned at the medium-speed storage layer, a total of 5TB of storage space was saved, and the cleaning operation took 1 hour. The cleaning report is stored in text format or structured data format and can be provided to the user through the user interface.

[0194] Preferably, the present invention further provides a storage control device based on big data, comprising an intelligent SSD controller, a storage level sensor, an SCM memory, a hierarchical storage medium, a system resource management module and a user interface, wherein the storage level sensor and the SCM memory are integrated inside the SSD controller, the hierarchical storage medium is connected to the intelligent SSD controller via a storage interface, the system resource management module is connected to the intelligent SSD controller via a system summary, and the user interface is connected to the system resource management module via a network interface, and is used to execute the storage control method based on big data as described above, wherein the intelligent SSD controller of the storage control device based on big data comprises:

[0195] The data dynamic association analysis module is used to obtain the original data stream and application requirement information; perform data dynamic association analysis on the original data stream according to the application requirement information to obtain a dynamically updated data association map; generate a feature vector for the dynamically updated data association map to obtain a data feature vector;

[0196] The dynamic value grading module is used to perform intelligent quality assessment on data feature vectors to obtain data quality labels; perform timeliness assessment based on data feature vectors and access heat assessment to obtain timeliness scores and access heat scores; obtain system resource load information; calculate application scenario weights based on data feature vectors and system resource load information to obtain application scenario weights; perform data value assessment based on timeliness scores, access heat scores, application scenario weights, and data quality labels to obtain data value scores; perform data strategy grading processing based on data value scores to obtain data value grade data;

[0197] An adaptive resource allocation module is used to perform adaptive resource allocation according to data value level data to obtain a resource allocation plan;

[0198] The intelligent data cleaning module is used to perform intelligent data cleaning according to the resource allocation plan and data quality labels to obtain a data cleaning report.

[0199] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0200] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A storage control method based on big data, characterized in that: The following steps are involved: Step S1: obtaining the original data stream and application requirement information; performing data dynamic correlation analysis on the original data stream according to the application requirement information to obtain a dynamically updated data correlation map; generating a feature vector for the dynamically updated data correlation map to obtain a data feature vector; Step S2: Perform intelligent quality assessment on the data feature vector to obtain a data quality label; perform timeliness assessment based on the data feature vector and perform access popularity assessment to obtain a timeliness score and an access popularity score; Obtain information about system resource load; calculate application scenario weights based on data feature vectors and system resource load information to obtain application scenario weights; evaluate data value based on timeliness score, access popularity score, application scenario weights, and data quality labels to obtain data value scores; perform data strategy grading based on data value scores to obtain data value grade data; Step S3: Adaptively allocate resources according to the data value level data to obtain a resource allocation plan, wherein step S3 is specifically as follows: Step S31: performing resource demand prediction on the data value level data to obtain a resource demand prediction result; Step S32: performing resource pool management according to the resource demand prediction result to obtain resource pool configuration data; Step S33: Formulate a resource allocation strategy based on the resource demand prediction result and the resource pool configuration data to obtain a resource allocation strategy; Step S34: Allocate resources according to the resource allocation strategy and the data value level data to obtain a resource allocation plan; Step S4: Perform intelligent data cleaning according to the resource allocation plan and data quality labels to obtain a data cleaning report to implement data storage control work, wherein step S4 is specifically as follows: Step S41: Formulate a cleanup strategy based on the resource allocation plan and the data quality label to obtain a data cleanup strategy; Step S42: Identify data according to the data cleaning strategy and data value level to obtain a list of data to be cleaned; Step S43: performing data cleaning on the data list to be cleaned according to the data cleaning strategy to obtain a cleaning result; Step S44: Generate a cleanup report based on the cleanup results to obtain a data cleanup report.

2. The storage control method based on big data according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: obtaining the original data stream and application requirement information; extracting metadata from the original data stream to obtain metadata; Step S12: extracting content features from the original data stream according to the metadata to obtain content features; Step S13: Analyze the access pattern according to the metadata to obtain the access pattern; Step S14: Perform data dynamic correlation analysis on metadata and content features according to access patterns and application demand information to obtain a dynamically updated data correlation graph; Step S15: Generate feature vectors for metadata, content features, access patterns, and dynamically updated data association graphs to obtain data feature vectors.

3. The storage control method based on big data according to claim 2, characterized in that: Step S14 includes the following steps: Step S141: performing data preprocessing on metadata and content features to obtain preprocessed data; Step S142: performing reference relationship identification on the pre-processed data to obtain a reference relationship set; Step S143: Calculate data similarity on the preprocessed data to obtain a similarity matrix; Step S144: performing time series relationship identification on the pre-processed data to obtain a time series relationship set; Step S145: constructing a data association graph according to the reference relationship set, the similarity matrix and the temporal relationship set to obtain a data association graph; Step S146: Perform dynamic association analysis on the data association graph according to the access mode and application requirement information to obtain a dynamically updated data association graph.

4. The storage control method based on big data according to claim 1, characterized in that: Step S2 includes the following steps: Step S21: performing feature vector preprocessing on the data feature vector to obtain a preprocessed feature vector; performing quality assessment on the preprocessed feature vector using a preset machine learning model to obtain a quality score; Step S22: Generate a data quality label according to the quality score to obtain a data quality label; Step S23: performing timeliness evaluation and access popularity evaluation based on the data feature vector to obtain a timeliness score and an access popularity score; Step S24: Obtain system resource load information; calculate the application scenario weight according to the data feature vector and the system resource load information to obtain the application scenario weight; Step S25: quantify the value according to the timeliness score, access popularity score, application scenario weight and data quality label to obtain a preliminary value score; standardize the preliminary value score to obtain a data value score; Step S26: Formulate a grading strategy based on the data value score, system resource load information, and application demand information to obtain a dynamic grading strategy; Step S27: Classify the data according to the dynamic classification strategy and data value score to obtain data value level data.

5. The storage control method based on big data according to claim 4 is characterized in that: Step S23 includes the following steps: Step S231: acquiring time features of the data feature vector to obtain time features; calculating time differences of the time features to obtain time difference values; Step S232: Calculate the initial timeliness score using the time difference and the preset time decay function to obtain the initial timeliness score; perform historical access frequency statistics on the data feature vector to obtain the historical access frequency; Step S233: adjusting the decay function parameters according to the historical access frequency and the initial timeliness score to obtain an adjusted decay function; calculating the final timeliness score using the adjusted decay function and the time difference to obtain a timeliness score; Step S234: acquiring access features of the data feature vector to obtain access features; calculating an initial heat score using the access features and a preset heat calculation method to obtain an initial heat score; Step S235: acquiring associated data from the data feature vector to obtain an associated data list; calculating associated data value according to the associated data list to obtain an associated data value list; Step S236: performing an associated data impact calculation based on the associated data value list and the associated data list to obtain an associated data impact value; performing a final heat score calculation on the associated data impact value and the initial heat score to obtain an access heat score.

6. The storage control method based on big data according to claim 4, characterized in that: Step S24 includes the following steps: Step S241: extracting scene-related features from the data feature vector to obtain scene-related features; Step S242: Identify the application scenario according to the scenario-related features to obtain an application scenario identifier; Step S243: performing initial scene weight analysis according to the application scene identifier using a preset scene weight table to obtain an initial scene weight; Step S244: obtaining user priority setting information; dynamically adjusting the initial scene weight according to the system resource load information and the user priority setting information to obtain the application scene weight.

7. The storage control method based on big data according to claim 4, characterized in that: Step S26 includes the following steps: Step S261: Determine the classification threshold according to the data value score, system resource load information and application demand information to obtain the classification threshold; Step S262: selecting a data storage medium according to the classification threshold and the data value score, and obtaining a storage medium matching table; Step S263: generating an initial classification strategy according to the classification threshold and the storage medium matching table to obtain an initial classification strategy; Step S264: dynamically optimize the initial grading strategy according to the storage resource load information and the application demand information to obtain a dynamic grading strategy.

8. A storage control device based on big data, characterized in that: The intelligent SSD controller comprises a storage-level sensor, an SCM memory, a hierarchical storage medium, a system resource management module and a user interface, wherein the storage-level sensor and the SCM memory are integrated inside the SSD controller, the hierarchical storage medium is connected to the intelligent SSD controller via a storage interface, the system resource management module is connected to the intelligent SSD controller via a system summary, and the user interface is connected to the system resource management module via a network interface, and is used to execute the storage control method based on big data as claimed in claim 1. The intelligent SSD controller of the storage control device based on big data comprises: The data dynamic association analysis module is used to obtain the original data stream and application requirement information; perform data dynamic association analysis on the original data stream according to the application requirement information to obtain a dynamically updated data association map; generate a feature vector for the dynamically updated data association map to obtain a data feature vector; The dynamic value grading module is used to perform intelligent quality assessment on data feature vectors to obtain data quality labels; perform timeliness assessment based on data feature vectors and access heat assessment to obtain timeliness scores and access heat scores; obtain system resource load information; calculate application scenario weights based on data feature vectors and system resource load information to obtain application scenario weights; perform data value assessment based on timeliness scores, access heat scores, application scenario weights, and data quality labels to obtain data value scores; perform data strategy grading processing based on data value scores to obtain data value grade data; An adaptive resource allocation module is used to perform adaptive resource allocation according to data value level data to obtain a resource allocation plan; The intelligent data cleaning module is used to perform intelligent data cleaning according to the resource allocation plan and data quality labels to obtain a data cleaning report.

Images