AI intelligent hierarchical storage management system for mass unstructured data
By deeply integrating distributed architecture with AI technology, the problems of inflexible access, inaccurate feature parsing, and lack of dynamic hierarchical decision-making in the management of massive unstructured data have been solved, achieving efficient, secure, and low-cost data storage management.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN ANRUIBO TECH CO LTD
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to effectively manage the entire lifecycle of massive amounts of unstructured data, resulting in problems such as inflexible data access, inaccurate feature parsing, stagnant hierarchical decision-making, wasted storage resources, and low security.
A distributed multi-protocol adaptation architecture is adopted to achieve parallel access of multi-source data. It combines deep learning with traditional feature extraction for hybrid parsing, dynamic hierarchical decision-making based on reinforcement learning, integrates a four-layer storage architecture, adopts asynchronous migration and LRU-K cache optimization, and implements layered security protection and distributed operation and maintenance monitoring.
It enables accurate feature extraction and dynamic hierarchical management of various data types, optimizes storage resource scheduling, improves data migration efficiency and security, reduces costs, and enhances system intelligence and operational efficiency.
Smart Images
Figure CN122152241A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing, and in particular to an AI-powered intelligent hierarchical storage management system for massive amounts of unstructured data. Background Technology
[0002] With the acceleration of digital transformation, massive amounts of unstructured data, such as images, videos, logs, and documents, are experiencing explosive growth. Traditional storage systems face challenges such as high storage costs, low retrieval efficiency, and uneven value density. Enterprises urgently need to achieve automatic hierarchical management of data through intelligent means to balance storage resource allocation with business needs. AI-powered intelligent hierarchical storage management systems have emerged to address this need.
[0003] Current methods on the market have significant shortcomings. They lack deep integration of AI technology and distributed architecture, making it difficult to adapt to the full lifecycle management needs of massive amounts of unstructured data. Most methods use a single access protocol, failing to achieve parallel access to multi-source data and lacking elastic scalability, easily creating "data silos" and struggling to handle high-concurrency access scenarios. They also lack effective data pre-validation and deduplication mechanisms, easily introducing invalid data. Feature parsing often uses single traditional algorithms, relying on manually designed features, resulting in weak semantic expression and poor generalization. It cannot accurately extract features from multiple data types and lacks standardized processing, making it difficult to support intelligent hierarchical classification. Hierarchical decision-making often uses fixed threshold models, failing to dynamically optimize based on data popularity and business needs, leading to wasted storage resources or low access efficiency. Storage scheduling and caching strategies are simple, lacking intelligent algorithms, making it difficult to balance access speed and storage costs. Data migration often uses traditional copying methods, prone to interruption, lacking breakpoint resumption capabilities, and interfering with business operations. Security protection and operation and maintenance monitoring rely on manual processes, lacking AI anomaly detection and dynamic permission control, resulting in significant security risks and low operation and maintenance efficiency. Summary of the Invention
[0004] To improve existing systems, an AI-powered intelligent hierarchical storage management system for massive amounts of unstructured data is provided. This method deeply integrates distributed architecture with AI technology to achieve accurate feature analysis and dynamic intelligent hierarchical classification, optimize storage resource scheduling, caching strategies and data migration efficiency, and is equipped with layered security protection and AI operation and maintenance monitoring, thus possessing high intelligence, high efficiency, low cost and high security.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: An AI-powered intelligent hierarchical storage management system for massive amounts of unstructured data, including: Unstructured data access module: It adopts a distributed multi-protocol adaptation architecture, supports parallel access of various types of unstructured data, and intercepts invalid data through load balancing, data pre-verification and preliminary deduplication, and supports elastic expansion of access nodes; AI Intelligent Feature Parsing Module: It adopts a hybrid parsing architecture that combines deep learning with traditional feature extraction. For text, image, audio and video data types, it uses dedicated parsing technology to extract features and generates a unified feature list after standardization processing. Dynamic hierarchical decision-making module: Based on reinforcement learning algorithm, an intelligent hierarchical decision-making model is built. The data hierarchical results are adjusted through weight analysis and access log statistics, and the hierarchical threshold is adaptively optimized. Multi-level storage resource scheduling module: integrates a four-layer storage architecture, monitors the status of each storage layer in real time, and allocates storage resources using a greedy algorithm based on the hierarchical results; Data migration module: It adopts an asynchronous migration architecture, schedules migration tasks according to priority, and combines breakpoint resume, data verification and bandwidth adjustment technologies to migrate data between storage layers. AI Cache Optimization Module: Based on the LRU-K algorithm, a cache model is built to cache frequently accessed data. The cache strategy is optimized through cache preloading, hit rate monitoring and expiration mechanism. Clustered deployment is used to improve cache response speed. Security Management Module: Employs layered security protection, and manages the entire data process through data encryption, RBAC access control, sensitive data anonymization, and security auditing; Operation and maintenance monitoring module: It adopts a distributed monitoring architecture to collect system operation indicators in real time, and identifies faults and issues alarms in a tiered manner through AI anomaly detection.
[0006] Preferably, the unstructured data access module specifically includes: Protocol adaptation unit: Adopts a distributed multi-protocol adaptation architecture, integrating FTP, SFTP, HTTP / HTTPS, SMB, NFS and object storage protocols, and is compatible with various unstructured data formats; Load balancing unit: Uses a round-robin scheduling algorithm to distribute various access requests, avoiding overload of a single access node; Data preprocessing unit: Identifies and intercepts invalid and malicious data through file header feature matching technology, and performs preliminary deduplication by combining file size, modification time and hash value triple comparison; Elastic expansion unit: Adds new access nodes based on dynamic changes in the amount of accessed data to adapt to scenarios with massive concurrent data access.
[0007] Preferably, the AI intelligent feature parsing module specifically includes: Data type adaptation unit: Formulate parsing strategies for the characteristics of different data types and build a hybrid parsing architecture that combines deep learning and traditional feature extraction; Feature extraction unit: For text, semantic features and keywords are extracted using the BERT model; for images, visual features are extracted using the CNN model and feature labels are generated; for audio, audio features are extracted using MFCC; and for video, features are extracted by combining keyframe extraction with the CNN model. Feature standardization unit: Performs unified standardization processing on various extracted feature data, normalizes feature values to a specified range, and integrates them to generate a feature list in a unified format; Feature association unit: Associates the standardized feature list with the original data ID, labels the basic data attributes, and completes the integration and archiving of feature data.
[0008] Preferably, the dynamic hierarchical decision-making module specifically includes: Tiered Standard Setting Unit: Based on data characteristics, access frequency, storage cost and business needs, four levels of storage standards are set for hot, warm, cold and archived data, clarifying the storage priority, access latency requirements and storage period for each level of data; Feature weight analysis unit: The hierarchical analysis method is used to assign weights to data features, access frequency, storage cost and business demand classification indicators, clarify the proportion of each indicator and quantify the classification basis; Log statistics and analysis unit: collects access log data in real time, and uses a sliding window algorithm to statistically analyze access frequency, access time distribution and access trends to obtain data popularity changes; Tiered threshold adaptive unit: Real-time monitoring of storage load, data growth rate and business demand changes, automatically optimizes tiered thresholds and dynamically adjusts data tiering results.
[0009] Preferably, the multi-level storage resource scheduling module specifically includes: Storage Architecture Integration Unit: Integrates high-speed SSD storage layer, SAS hard disk storage layer, SATA hard disk storage layer and object storage layer, obtains the data types adapted to each layer, and builds a layered storage architecture; Resource status monitoring unit: collects the remaining capacity, read / write speed, and response latency of each storage layer in real time, and generates storage resource status reports; Resource allocation and scheduling unit: Based on the dynamic hierarchical decision results, a greedy algorithm is used to allocate data of different levels to the corresponding storage layer, and to balance access speed and storage cost; Elastic expansion unit: Real-time monitoring of the capacity of each storage layer. When the capacity of a certain layer reaches the threshold, an expansion command is triggered to add storage nodes and integrate them into the storage cluster.
[0010] Preferably, the data migration module specifically includes: Migration task scheduling unit: It adopts a priority scheduling algorithm, combined with data hierarchical changes and storage resource status, to prioritize the scheduling of migration tasks that affect business access, and delay the scheduling of cold data and archived data. Data migration execution unit: adopts an asynchronous migration architecture, has built-in breakpoint resume technology, records migration progress indicators, and resumes migration from the breakpoint after recovery in interrupted scenarios; Migration verification unit: After migration is completed, it verifies data integrity through hash value comparison technology, monitors abnormal situations during the migration process, and reports the migration status. Bandwidth control unit: Monitors service load and network bandwidth status in real time and dynamically adjusts migration bandwidth.
[0011] Preferably, the AI cache optimization module specifically includes: Cache model building unit: Based on the LRU-K algorithm, an intelligent cache model is built. Combining data classification results and access logs, hot data and frequently accessed warm data are filtered and cached in the SSD cache cluster. Cache strategy optimization unit: Adopts cache preloading technology, combined with data access trends, loads data that will be accessed soon into cache nodes, and sets differentiated cache expiration times for different levels of data according to data classification and access frequency; Cache monitoring unit: Real-time statistics on cache hit rate. When the hit rate is lower than the set threshold, the reason for cache failure is obtained and the cache strategy is adjusted.
[0012] Preferably, the security control module specifically includes: Data encryption unit: It adopts a layered encryption strategy, encrypting data during transmission using the TLS1.3 protocol, and statically encrypting data in the storage state using the AES-256 algorithm; Access control unit: Based on the RBAC permission model, it assigns access permissions to different users, down to specific operations such as data viewing, modification, deletion, and migration, and records all users' access behavior logs completely; Sensitive data desensitization unit: By identifying sensitive information in the data, sensitive content is processed using methods such as masking and replacement; Security Audit Unit: Regularly conducts comprehensive audits of system security status, user access logs, and data encryption status to identify security risks and trigger tiered alarms upon discovering problems.
[0013] Preferably, the operation and maintenance monitoring module specifically includes: Monitoring data acquisition unit: Adopts a distributed monitoring architecture to collect real-time operating indicators such as CPU utilization, memory usage, network bandwidth, storage capacity, migration speed, and cache hit rate; AI Anomaly Detection Unit: Based on the Isolation Forest algorithm, an anomaly detection model is built. By comparing historical and real-time data, node failures, network interruptions, storage overloads, and migration failures are identified and classified into three levels: critical, severe, and general anomalies. Operation and maintenance management unit: Supports remote management of storage nodes and access nodes, performs node restart, configuration modification, and fault diagnosis operations, and records operation and maintenance operations.
[0014] Compared with the prior art, the advantages of the present invention are: The core advantages are formed by the deep integration of distributed architecture and AI technology. Relying on a distributed multi-protocol adaptable architecture, it enables parallel and elastic access to multi-source unstructured data. Combined with data pre-verification and preliminary deduplication mechanisms, it effectively intercepts invalid data, improving data quality and access efficiency from the source. Employing a hybrid parsing architecture combining deep learning and traditional feature extraction, it can accurately extract features from various data types such as text, images, and audio / video, and standardize them, laying a solid data foundation for hierarchical decision-making. Based on reinforcement learning and the analytic hierarchy process (AHP), a dynamic hierarchical model is built, which can adaptively optimize hierarchical thresholds and indicator weights, breaking through the limitations of traditional fixed hierarchical models. Integrating a four-layer storage architecture, combined with greedy algorithms and LRU-K caching optimization, it achieves intelligent scheduling of storage resources and improved access performance, balancing business efficiency and storage costs. The asynchronous migration architecture, combined with breakpoint resume and dynamic bandwidth adjustment technologies, ensures efficient and stable data migration without interfering with business operations. Layered security protection and distributed AI operation and maintenance monitoring enable end-to-end data encryption, access control, and intelligent early warning of system failures. Overall, it features high adaptability, high intelligence, high efficiency, low cost, high security, and easy operation and maintenance, and can fully support intelligent hierarchical storage management of massive unstructured data throughout its entire lifecycle. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the system proposed in this invention; Figure 2 This is a diagram of the unstructured data access module proposed in this invention; Figure 3 This is a diagram of the AI intelligent feature parsing module proposed in this invention; Figure 4 This is a diagram of the dynamic hierarchical decision-making module proposed in this invention; Figure 5 This is a diagram of the multi-level storage resource scheduling module proposed in this invention; Figure 6 This is a diagram of the data migration module proposed in this invention; Figure 7 This is a diagram of the AI caching optimization module proposed in this invention; Figure 8 This is a diagram of the safety management module proposed in this invention; Figure 9This is a diagram of the operation and maintenance monitoring module proposed in this invention. Detailed Implementation
[0016] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.
[0017] See Figure 1 As shown, the AI-powered intelligent hierarchical storage management system for massive amounts of unstructured data includes: Unstructured data access module: It adopts a distributed multi-protocol adaptation architecture, supports parallel access of various types of unstructured data, and intercepts invalid data through load balancing, data pre-verification and preliminary deduplication, and supports elastic expansion of access nodes; AI Intelligent Feature Parsing Module: It adopts a hybrid parsing architecture that combines deep learning with traditional feature extraction. For text, image, audio and video data types, it uses dedicated parsing technology to extract features and generates a unified feature list after standardization processing. Dynamic hierarchical decision-making module: Based on reinforcement learning algorithm, an intelligent hierarchical decision-making model is built. The data hierarchical results are adjusted through weight analysis and access log statistics, and the hierarchical threshold is adaptively optimized. Multi-level storage resource scheduling module: integrates a four-layer storage architecture, monitors the status of each storage layer in real time, and allocates storage resources using a greedy algorithm based on the hierarchical results; Data migration module: It adopts an asynchronous migration architecture, schedules migration tasks according to priority, and combines breakpoint resume, data verification and bandwidth adjustment technologies to migrate data between storage layers. AI Cache Optimization Module: Based on the LRU-K algorithm, a cache model is built to cache frequently accessed data. The cache strategy is optimized through cache preloading, hit rate monitoring and expiration mechanism. Clustered deployment is used to improve cache response speed. Security Management Module: Employs layered security protection, and manages the entire data process through data encryption, RBAC access control, sensitive data anonymization, and security auditing; Operation and maintenance monitoring module: It adopts a distributed monitoring architecture to collect system operation indicators in real time, and identifies faults and issues alarms in a tiered manner through AI anomaly detection.
[0018] See Figure 2 As shown, the unstructured data access module specifically includes: Protocol adaptation unit: Adopts a distributed multi-protocol adaptation architecture, integrating FTP, SFTP, HTTP / HTTPS, SMB, NFS and object storage protocols, and is compatible with various unstructured data formats; Load balancing unit: Uses a round-robin scheduling algorithm to distribute various access requests, avoiding overload of a single access node; Data preprocessing unit: Identifies and intercepts invalid and malicious data through file header feature matching technology, and performs preliminary deduplication by combining file size, modification time and hash value triple comparison; Elastic expansion unit: Adds new access nodes based on dynamic changes in the amount of accessed data to adapt to scenarios with massive concurrent data access.
[0019] Specifically, a pluggable multi-protocol adaptation architecture is adopted to build a protocol-independent communication contract. Multi-protocol compatibility is achieved through four layers of collaborative abstraction, including the transport layer and protocol semantic layer. Independent parsing plugins are deployed for each access protocol, such as FTP, SFTP, and HTTP / HTTPS. Each plugin implements protocol parsing and serialization through a unified interface. Protocol adaptation can be added without modifying the core code, improving scalability. A multi-format compatibility module is built in. By using a preset file header feature library of common unstructured data, the system automatically identifies the data format and converts different data formats into the standard format supported by the system, enabling parallel access of multi-source data and avoiding access failures caused by format incompatibility. The pre-verification uses a two-factor authentication method combining file extension and header features. It first quickly filters files with extensions that do not meet the requirements, then reads the first 4-8 bytes of the file's Magic Number and compares it with a preset feature library to intercept invalid data with altered formats. The deduplication process uses the SHA256 hash algorithm, which processes files through streaming computation. It reads data in blocks to generate hash digests, eliminating the need to load the entire file into memory and improving processing efficiency. At the same time, it compares file sizes to assist in deduplication. Files with the same hash digest are then compared by comparing the first and last 64KB of data to avoid false positives caused by hash collisions.
[0020] See Figure 3 As shown, the AI intelligent feature parsing module specifically includes: Data type adaptation unit: Formulate parsing strategies for the characteristics of different data types and build a hybrid parsing architecture that combines deep learning and traditional feature extraction; Feature extraction unit: For text, semantic features and keywords are extracted using the BERT model; for images, visual features are extracted using the CNN model and feature labels are generated; for audio, audio features are extracted using MFCC; and for video, features are extracted by combining keyframe extraction with the CNN model. Feature standardization unit: Performs unified standardization processing on various extracted feature data, normalizes feature values to a specified range, and integrates them to generate a feature list in a unified format; Feature association unit: Associates the standardized feature list with the original data ID, labels the basic data attributes, and completes the integration and archiving of feature data.
[0021] Specifically, for text data, a BERT pre-trained model is used to extract semantic features and capture text contextual relationships. The TF-IDF algorithm is combined to extract surface features such as keywords and thematic terms, providing double verification of the core text information. For image data, a CNN convolutional neural network is used to extract deep visual features such as texture, contour, and color. The SIFT algorithm is then used to extract local feature points, taking into account both global and local features. For audio data, Mel-frequency cepstral coefficients are used to extract features such as timbre, rhythm, and spectrum. Short-time Fourier transform is combined to capture audio temporal features. For video data, based on frame image feature extraction, a temporal convolutional network is used to extract inter-frame correlation features and capture dynamic changes in the video. Since different types of data have different feature dimensions and formats, standardization is required to eliminate the influence of units. The Min-Max normalization method is used to map the numerical values of various features to a unified range and unify the data type of features. The extracted features are screened to remove redundant and invalid features and retain the core features related to hierarchical storage, such as access frequency correlation features and data value features. Non-numerical features are converted into numerical features through feature encoding technology. Finally, all effective features are integrated to generate a standardized and structured unified feature list.
[0022] See Figure 4 As shown, the dynamic hierarchical decision-making module specifically includes: Tiered Standard Setting Unit: Based on data characteristics, access frequency, storage cost and business needs, four levels of storage standards are set for hot, warm, cold and archived data, clarifying the storage priority, access latency requirements and storage period for each level of data; Feature weight analysis unit: The hierarchical analysis method is used to assign weights to data features, access frequency, storage cost and business demand classification indicators, clarify the proportion of each indicator and quantify the classification basis; Log statistics and analysis unit: collects access log data in real time, and uses a sliding window algorithm to statistically analyze access frequency, access time distribution and access trends to obtain data popularity changes; Tiered threshold adaptive unit: Real-time monitoring of storage load, data growth rate and business demand changes, automatically optimizes tiered thresholds and dynamically adjusts data tiering results.
[0023] Specifically, a core decision-making model is built based on reinforcement learning algorithms. Combined with historical storage data, access logs and business requirements, the model is trained and parameters are calibrated. The initial judgment criteria for four levels of storage—hot, warm, cold and archive—are defined, and the core judgment dimensions for each level of storage are preset, including five core indicators: data access frequency, access interval, data value, storage duration and business priority. A weight analysis system is built, and the initial weights of each indicator are set according to business scenario requirements to ensure that the classification results are consistent with actual business needs. It connects to the unified feature list output by the AI intelligent feature analysis module to extract core features such as data value and format type. At the same time, it collects system access logs in real time to collect access features such as access frequency, recent access time, and access terminal. It adopts a dynamic weight allocation mechanism, abandoning the fixed weight mode. Through reinforcement learning algorithms, it analyzes the impact of each indicator on the classification result in real time and automatically adjusts the indicator weights. For example, in high-frequency access scenarios, the access frequency weight is automatically increased; in core business data scenarios, the data value weight is given priority to ensure that the weight allocation is dynamically matched with the business. Based on the initial four-level storage standard and dynamically adjusted indicator weights, the input feature data is comprehensively evaluated. A multi-dimensional threshold comparison method is used to compare each data feature with preset thresholds one by one. Combined with the weighted calculation results, the storage level of the data is determined: frequently accessed, recently accessed, and high-value data are classified as hot data; data with moderate access frequency and some business value are classified as warm data; data with low access frequency and long storage time are classified as cold data; and data with no long-term access and no immediate business value is classified as archived data. Establish a tiered result optimization mechanism to monitor changes in data access behavior and adjustments to business needs in real time, and regularly review the tiered data; if the access frequency of a certain type of data changes abruptly, the system will automatically trigger a tier adjustment, recalculate the weights and thresholds, and adjust it to the corresponding level; through reinforcement learning algorithms, continuously learn from historical adjustment data and business feedback, and adaptively optimize the judgment thresholds of each level of storage.
[0024] See Figure 5 As shown, the multi-level storage resource scheduling module specifically includes: Storage Architecture Integration Unit: Integrates high-speed SSD storage layer, SAS hard disk storage layer, SATA hard disk storage layer and object storage layer, obtains the data types adapted to each layer, and builds a layered storage architecture; Resource status monitoring unit: collects the remaining capacity, read / write speed, and response latency of each storage layer in real time, and generates storage resource status reports; Resource allocation and scheduling unit: Based on the dynamic hierarchical decision results, a greedy algorithm is used to allocate data of different levels to the corresponding storage layer, and to balance access speed and storage cost; Elastic expansion unit: Real-time monitoring of the capacity of each storage layer. When the capacity of a certain layer reaches the threshold, an expansion command is triggered to add storage nodes and integrate them into the storage cluster.
[0025] Specifically, the system integrates a four-layer storage architecture consisting of SSDs, SAS hard drives, SATA hard drives, and tape libraries, clearly defining the storage positioning and suitable scenarios for each layer: the SSD layer is used to store hot data, ensuring extremely fast response for high-frequency access; the SAS layer stores warm data, balancing access speed and storage cost; the SATA layer stores cold data, focusing on large-capacity, low-cost storage; and the tape library is used for archived data, achieving long-term, low-cost retention. During the integration process, a distributed architecture is used to network the nodes of each storage layer, and interconnection between layers is achieved through a unified storage interface encapsulation. At the same time, the storage parameters of each layer are initialized, a unified storage resource pool is established, and the resources of all storage nodes are included in pooled management. A distributed monitoring agent is deployed on each storage node to collect core operating metrics of each storage layer in real time, including capacity utilization, read / write response time, CPU load, bandwidth utilization, and node failure rate. The monitoring data is synchronized to the scheduling center through a message queue, where the scheduling center aggregates and analyzes the data to establish a visual ledger of storage status. At the same time, a warning threshold is preset for each metric. When the capacity utilization of a storage layer exceeds 80%, the response time expires, or a node fails, a warning is immediately triggered. The scheduling module connects to the hierarchical decision-making module and uses a greedy algorithm to achieve optimal resource allocation: hot data is prioritized for allocation to the SSD layer to ensure access efficiency by leveraging its high-speed read / write characteristics; warm data is allocated to the SAS layer to balance speed and cost; cold data is allocated to the SATA layer to maximize the advantages of large capacity and low cost; and archived data is allocated to the tape library for long-term storage. During the allocation process, the remaining resources and load of each storage layer are compared in real time. If the target storage layer is overloaded, some data is automatically scheduled to an idle storage layer. At the same time, based on business priorities, the hierarchical data of core business is given priority in allocating high-quality storage resources.
[0026] See Figure 6 As shown, the data migration module specifically includes: Migration task scheduling unit: It adopts a priority scheduling algorithm, combined with data hierarchical changes and storage resource status, to prioritize the scheduling of migration tasks that affect business access, and delay the scheduling of cold data and archived data. Data migration execution unit: adopts an asynchronous migration architecture, has built-in breakpoint resume technology, records migration progress indicators, and resumes migration from the breakpoint after recovery in interrupted scenarios; Migration verification unit: After migration is completed, it verifies data integrity through hash value comparison technology, monitors abnormal situations during the migration process, and reports the migration status. Bandwidth control unit: Monitors service load and network bandwidth status in real time and dynamically adjusts migration bandwidth.
[0027] Specifically, a priority scheduling mechanism is adopted, which sets migration priorities based on business priorities and data popularity: core business data and frequently accessed hot data have the highest priority and are allocated migration resources first; archived data and non-core cold data have lower priority and are migrated in staggered shifts. A migration task queue is built, and all migration tasks are sorted and managed. A queue rate limiting strategy is adopted to prevent bandwidth congestion and system overload caused by concurrent migration of multiple tasks. A built-in dynamic task priority adjustment function is also provided. If a high-priority task is urgent, low-priority tasks can be interrupted to complete the high-priority migration first. An asynchronous migration architecture is adopted, ensuring that the migration process does not affect normal access to and read / write operations of the source data. During migration, the data is first sharded, dividing large amounts of data into fixed-size shards and using streaming transmission to reduce memory usage. Combined with dynamic bandwidth adjustment technology, the system bandwidth usage is monitored in real time. When business access bandwidth is tight, the migration bandwidth is reduced, and when bandwidth is idle, the migration bandwidth is increased, balancing migration efficiency and business access experience. A breakpoint resume technology is adopted, which records the sharding progress and checksum of the migrated data. If the migration is interrupted, it is not necessary to re-migrate all the data after recovery; only the unfinished shards are resumed.
[0028] See Figure 7 As shown, the AI caching optimization module specifically includes: Cache model building unit: Based on the LRU-K algorithm, an intelligent cache model is built. Combining data classification results and access logs, hot data and frequently accessed warm data are filtered and cached in the SSD cache cluster. Cache strategy optimization unit: Adopts cache preloading technology, combined with data access trends, loads data that will be accessed soon into cache nodes, and sets differentiated cache expiration times for different levels of data according to data classification and access frequency; Cache monitoring unit: Real-time statistics on cache hit rate. When the hit rate is lower than the set threshold, the reason for cache failure is obtained and the cache strategy is adjusted.
[0029] Specifically, the module builds a core cache model based on the LRU-K algorithm, defines the standard for setting the K value, and divides the cache into hot cache areas and ordinary cache areas. The hot cache area is used to store frequently accessed hot data to ensure extremely fast response; the ordinary cache area stores data with moderate access frequency. It adopts a clustered deployment mode, networks multiple cache nodes, and achieves load sharing among cache nodes through load balancing configuration. It initializes the cache capacity threshold and builds a cache data index system. The LRU-K elimination weight formula is: ; in, Let t be the LRU-K weights. The recent access count threshold. For the current time, This is the timestamp of the kth access. The module connects to the dynamic hierarchical decision-making module to obtain hierarchical results and system access logs, and statistically analyzes access frequency, access interval, and business priority in real time. It sets cache filtering thresholds to filter out hot data with access frequency reaching the threshold and high business priority, and prioritizes it for inclusion in the hot data cache area. Data with medium access frequency is included in the ordinary cache area, while cold data with low access frequency and archived data are not included in the cache. It realizes intelligent preloading function, and by analyzing historical access logs, it can discover data access patterns and preload related data into the cache before the business peak or after the first access to data, reserve hot data in advance, reduce the probability of cache missing, and improve access response speed. Real-time monitoring of cache operation status, including cache hit rate, cache capacity utilization, and load of each node, dynamically adjusting cache strategy; when the cache hit rate is lower than the preset threshold, optimizing the filtering threshold and K value, re-filtering high-frequency data, cleaning up data that has not been accessed for a long time or whose access frequency has decreased, and releasing cache resources; when the cache capacity utilization is close to the threshold, prioritizing the cleaning of inefficient data in the ordinary cache area and retaining data in the hot cache area. To address common anomalies during cache operation, a robust fault tolerance mechanism is established: to handle cache breakdown, a mutual exclusion lock is employed; to handle cache avalanche, a staggered cache expiration time strategy is adopted, setting different expiration times for different cached data; when a cache node fails, the clustered architecture automatically migrates the cache tasks of the failed node to a backup node, and quickly restores the cache service through data backup.
[0030] See Figure 8 As shown, the security management module specifically includes: Data encryption unit: It adopts a layered encryption strategy, encrypting data during transmission using the TLS1.3 protocol, and statically encrypting data in the storage state using the AES-256 algorithm; Access control unit: Based on the RBAC permission model, it assigns access permissions to different users, down to specific operations such as data viewing, modification, deletion, and migration, and records all users' access behavior logs completely; Sensitive data desensitization unit: By identifying sensitive information in the data, sensitive content is processed using methods such as masking and replacement; Security Audit Unit: Regularly conducts comprehensive audits of system security status, user access logs, and data encryption status to identify security risks and trigger tiered alarms upon discovering problems.
[0031] Specifically, a layered security architecture is adopted, clearly defining the four-level protection boundaries of the access layer, storage layer, application layer, and audit layer. A dual encryption mechanism of "transmission encryption + storage encryption" is used to achieve end-to-end data encryption. Transmission encryption uses the SSL / TLS protocol, and the encryption process is automatically triggered during all data interactions between modules, external data access, and data migration processes. The identities of the communicating parties are confirmed through two-way certificate verification. Storage encryption adopts a partitioned encryption method, setting up independent encryption partitions for data of different levels. The keys are uniformly managed by a dedicated key management module and are automatically changed periodically. Based on the RBAC access control model, a three-tier access control system of "administrator - operation and maintenance personnel - ordinary users" is constructed, and the access permission boundaries of each role are clearly defined. The permission allocation adopts the "principle of least privilege", assigning precise permissions to each user according to business needs, while realizing dynamic adjustment of permissions. When a user role changes, the permission configuration is automatically updated, and redundant permissions are cleaned up regularly.
[0032] See Figure 9 As shown, the operation and maintenance monitoring module specifically includes: Monitoring data acquisition unit: Adopts a distributed monitoring architecture to collect real-time operating indicators such as CPU utilization, memory usage, network bandwidth, storage capacity, migration speed, and cache hit rate; AI Anomaly Detection Unit: Based on the Isolation Forest algorithm, an anomaly detection model is built. By comparing historical and real-time data, node failures, network interruptions, storage overloads, and migration failures are identified and classified into three levels: critical, severe, and general anomalies. Operation and maintenance management unit: Supports remote management of storage nodes and access nodes, performs node restart, configuration modification, and fault diagnosis operations, and records operation and maintenance operations.
[0033] Specifically, the aggregated monitoring data is analyzed in real time, employing a dual detection method of "threshold comparison + trend prediction": on the one hand, real-time data is compared with preset thresholds to identify anomalies such as indicators exceeding limits; on the other hand, historical operational data is analyzed to predict indicator trends and identify potential faults in advance, such as continuously increasing storage capacity or gradually extending module response time. After anomaly detection, alarms are categorized into three levels based on severity: emergency alarms are notified to operations and maintenance personnel via SMS and platform pop-ups, triggering emergency handling procedures; general alarms are sent to the platform to remind operations and maintenance personnel; and alert alarms are logged, requiring no immediate processing, and automatically associated with abnormal data and corresponding modules and nodes.
[0034] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0035] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.
[0036] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An AI-powered intelligent hierarchical storage management system for massive amounts of unstructured data, characterized in that: include: Unstructured data access module: It adopts a distributed multi-protocol adaptation architecture, supports parallel access of various types of unstructured data, and intercepts invalid data through load balancing, data pre-verification and preliminary deduplication, and supports elastic expansion of access nodes; AI Intelligent Feature Parsing Module: It adopts a hybrid parsing architecture that combines deep learning with traditional feature extraction. For text, image, audio and video data types, it uses dedicated parsing technology to extract features and generates a unified feature list after standardization processing. Dynamic hierarchical decision-making module: Based on reinforcement learning algorithm, an intelligent hierarchical decision-making model is built. The data hierarchical results are adjusted through weight analysis and access log statistics, and the hierarchical threshold is adaptively optimized. Multi-level storage resource scheduling module: integrates a four-layer storage architecture, monitors the status of each storage layer in real time, and allocates storage resources using a greedy algorithm based on the hierarchical results; Data migration module: It adopts an asynchronous migration architecture, schedules migration tasks according to priority, and combines breakpoint resume, data verification and bandwidth adjustment technologies to migrate data between storage layers; AI Cache Optimization Module: Based on the LRU-K algorithm, a cache model is built to cache frequently accessed data. The cache strategy is optimized through cache preloading, hit rate monitoring and expiration mechanism. Clustered deployment is used to improve cache response speed. Security Management Module: Employs layered security protection, and manages the entire data process through data encryption, RBAC access control, sensitive data anonymization, and security auditing; Operation and maintenance monitoring module: It adopts a distributed monitoring architecture to collect system operation indicators in real time, and identifies faults and issues alarms in a tiered manner through AI anomaly detection.
2. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The unstructured data access module specifically includes: Protocol adaptation unit: Adopts a distributed multi-protocol adaptation architecture, integrating FTP, SFTP, HTTP / HTTPS, SMB, NFS and object storage protocols, and is compatible with various unstructured data formats; Load balancing unit: Uses a round-robin scheduling algorithm to distribute various access requests, avoiding overload of a single access node; Data preprocessing unit: Identifies and intercepts invalid and malicious data through file header feature matching technology, and performs preliminary deduplication by combining file size, modification time and hash value triple comparison; Elastic expansion unit: Adds new access nodes based on dynamic changes in the amount of accessed data to adapt to scenarios with massive concurrent data access.
3. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The AI intelligent feature parsing module specifically includes: Data type adaptation unit: Formulate parsing strategies for the characteristics of different data types and build a hybrid parsing architecture that combines deep learning and traditional feature extraction; Feature extraction unit: For text, semantic features and keywords are extracted using the BERT model; for images, visual features are extracted using the CNN model and feature labels are generated; for audio, audio features are extracted using MFCC; and for video, features are extracted by combining keyframe extraction with the CNN model. Feature standardization unit: Performs unified standardization processing on various extracted feature data, normalizes feature values to a specified range, and integrates them to generate a feature list in a unified format; Feature association unit: Associates the standardized feature list with the original data ID, labels the basic data attributes, and completes the integration and archiving of feature data.
4. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The dynamic hierarchical decision-making module specifically includes: Tiered Standard Setting Unit: Based on data characteristics, access frequency, storage cost and business needs, four levels of storage standards are set for hot, warm, cold and archived data, clarifying the storage priority, access latency requirements and storage period for each level of data; Feature weight analysis unit: The hierarchical analysis method is used to assign weights to data features, access frequency, storage cost and business demand classification indicators, clarify the proportion of each indicator and quantify the classification basis; Log statistics and analysis unit: collects access log data in real time, and uses a sliding window algorithm to statistically analyze access frequency, access time distribution and access trends to obtain data popularity changes; Tiered threshold adaptive unit: Real-time monitoring of storage load, data growth rate and business demand changes, automatically optimizes tiered thresholds and dynamically adjusts data tiering results.
5. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The multi-level storage resource scheduling module specifically includes: Storage Architecture Integration Unit: Integrates high-speed SSD storage layer, SAS hard disk storage layer, SATA hard disk storage layer and object storage layer, obtains the data types adapted to each layer, and builds a layered storage architecture; Resource status monitoring unit: collects the remaining capacity, read / write speed, and response latency of each storage layer in real time, and generates storage resource status reports; Resource allocation and scheduling unit: Based on the dynamic hierarchical decision results, a greedy algorithm is used to allocate data of different levels to the corresponding storage layer, and to balance access speed and storage cost; Elastic expansion unit: Real-time monitoring of the capacity of each storage layer. When the capacity of a certain layer reaches the threshold, an expansion command is triggered to add storage nodes and integrate them into the storage cluster.
6. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The data migration module specifically includes: Migration task scheduling unit: It adopts a priority scheduling algorithm, combined with data hierarchical changes and storage resource status, to prioritize the scheduling of migration tasks that affect business access, and delay the scheduling of cold data and archived data. Data migration execution unit: adopts an asynchronous migration architecture, has built-in breakpoint resume technology, records migration progress indicators, and resumes migration from the breakpoint after recovery in interrupted scenarios; Migration verification unit: After migration is completed, it verifies data integrity through hash value comparison technology, monitors for abnormal situations during the migration process, and provides feedback on the migration status; Bandwidth control unit: Monitors service load and network bandwidth status in real time and dynamically adjusts migration bandwidth.
7. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The AI cache optimization module specifically includes: Cache model building unit: Based on the LRU-K algorithm, an intelligent cache model is built. Combining data classification results and access logs, hot data and frequently accessed warm data are filtered and cached in the SSD cache cluster. Cache strategy optimization unit: Adopts cache preloading technology, combined with data access trends, loads data that will be accessed soon into cache nodes, and sets differentiated cache expiration times for different levels of data according to data classification and access frequency; Cache monitoring unit: Real-time statistics on cache hit rate. When the hit rate is lower than the set threshold, the unit obtains the reason for cache failure and adjusts the cache strategy accordingly.
8. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The security management module specifically includes: Data encryption unit: It adopts a layered encryption strategy, encrypting data during transmission using the TLS1.3 protocol, and statically encrypting data in the storage state using the AES-256 algorithm; Access control unit: Based on the RBAC permission model, it assigns access permissions to different users, down to specific operations such as data viewing, modification, deletion, and migration, and records all users' access behavior logs completely; Sensitive data desensitization unit: By identifying sensitive information in the data, sensitive content is processed using methods such as masking and replacement; Security Audit Unit: Regularly conducts comprehensive audits of system security status, user access logs, and data encryption status to identify security risks and trigger tiered alarms upon discovering problems.
9. The AI-powered intelligent hierarchical storage management system for massive unstructured data according to claim 1, characterized in that, The operation and maintenance monitoring module specifically includes: Monitoring data acquisition unit: Adopts a distributed monitoring architecture to collect real-time operating indicators such as CPU utilization, memory usage, network bandwidth, storage capacity, migration speed, and cache hit rate; AI Anomaly Detection Unit: Based on the Isolation Forest algorithm, an anomaly detection model is built. By comparing historical and real-time data, node failures, network interruptions, storage overloads, and migration failures are identified and classified into three levels: critical, severe, and general anomalies. Operation and maintenance management unit: Supports remote management of storage nodes and access nodes, performs node restart, configuration modification, and fault diagnosis operations, and records operation and maintenance operations.