Data grading processing method based on storage manager

Abstract

The invention discloses a data grading processing method based on a storage manager, and the method comprises the steps: obtaining a multi-dimensional dynamic feature vector through refined perception and feature extraction; performing hierarchical value comprehensive prediction by using a preset fusion model, and predicting a future access trend; collaborative decision-making and scheduling of resource awareness are carried out based on grading value scores; and carrying out atomization non-interference migration and effect feedback through a migration plan and an intelligent scheduling scheme. According to the method, through triple mechanisms of resource pool isolation, self-adaptive rate control and intelligent opportunity scheduling, it is ensured that data migration operation does not generate perceptible performance interference on foreground key services under any load condition, and the strict requirements of high-performance application for stability and low delay are met; and a feedback optimization closed loop is introduced to autonomously learn from historical decision results and dynamically adjust a prediction model, a decision threshold and a scheduling strategy, so that the system can continuously adapt to a complex and changeable service environment, long-term optimal operation is realized, and tedious manual adjustment and optimization are avoided.

Description

Technical Field

[0001] This invention belongs to the field of computer storage technology, and specifically relates to a data hierarchical processing method based on a storage manager. Background Technology

[0002] With the explosive growth of data volume, it is difficult for a single type of storage medium to achieve the best balance between performance, capacity, and cost. Tiered storage architecture has emerged to address this, which uses high-speed, low-capacity storage media (such as Optane and NVMe SSDs) as the high-performance layer, and low-speed, high-capacity storage media (such as SATA SSDs, HDDs, and tapes) as the high-capacity or archive layer.

[0003] Traditional technologies rely solely on historical access frequency or time, making it difficult to accurately predict future access patterns. For data with periodic, bursty, or trend-based access characteristics, such strategies often perform poorly, resulting in "hot data" remaining in slow layers or "cold data" occupying valuable space in high-speed layers. Migration is typically performed on a file-by-file or fixed-size data block basis, lacking flexibility. Only a small portion of data in a file may be "hot," and migrating the entire file causes unnecessary I / O overhead and wasted space. Traditional solutions often focus only on data activity within a single storage volume or host. In distributed or virtualized environments, they cannot identify truly important hot data from a global perspective (e.g., across nodes, across virtual machines) and coordinate resource scheduling. Data migration (especially degradation migration) operations compete with foreground application I / O for storage bandwidth and computing resources, potentially causing application performance fluctuations.

[0004] In summary, existing technologies suffer from rigid strategies, slow response times, coarse hierarchical granularity, and a lack of global resource coordination. Summary of the Invention

[0005] To address the aforementioned problems in the existing technology, this invention provides a data hierarchical processing method based on a storage manager. The technical problem to be solved by this invention is achieved through the following technical solution: In a first aspect, the present invention provides a data hierarchical processing method based on a storage manager, the method comprising: Within each cycle, continuously monitor all I / O requests of the storage manager and extract multi-dimensional dynamic feature vectors of data units in the monitoring storage manager; Based on multidimensional dynamic feature vectors, the hierarchical value of each data unit is calculated using a pre-defined fusion model, and the hierarchical value score of each data unit is obtained. Using a policy scheduler, migration plans and intelligent scheduling schemes are generated based on hierarchical value scores, real-time load of the storage system, and storage layer status of the storage manager. Based on migration plans and intelligent scheduling schemes, the data of the data units to be migrated is migrated atomically and without interference to maintain data consistency; Perform predictive analysis on key metrics of the storage manager in the current cycle, and adaptively optimize the preset fusion model and policy scheduler for the next cycle.

[0006] In one embodiment of the present invention, continuous monitoring of all I / O of the storage manager and extraction of multi-dimensional dynamic feature vectors of data units in the monitoring storage manager include: Continuously monitor all I / O requests from the storage manager, capture metadata of all read and write requests, and generate access event streams; Real-time analysis of access event streams extracts multi-dimensional dynamic feature vectors for each data unit.

[0007] In one embodiment of the present invention, the multidimensional dynamic feature vector includes: Recent popularity features, temporal pattern features, spatial correlation features, and business semantic features.

[0008] In one embodiment of the present invention, based on a multidimensional dynamic feature vector, a preset fusion model is used to calculate the hierarchical value of each data unit, thereby obtaining a hierarchical value score for each data unit, including: Using a pre-defined fusion model, short-term value components, long-term value components, related value components, and cost value components are obtained based on multi-dimensional dynamic feature vectors. The hierarchical value score of each data unit is calculated by comprehensively considering the short-term value component, long-term value component, related value component, and cost value component, with dynamic weights.

[0009] In one embodiment of the present invention, a policy scheduler is used to generate a migration plan and an intelligent scheduling scheme based on hierarchical value scoring, the real-time load of the storage system, and the storage layer status of the storage manager, including: Using a policy scheduler, performance layers in data units with tiered value scores below a preset downgrade threshold are marked as downgrade candidates, and capacity layers in data units with tiered value scores above a preset upgrade threshold are marked as upgrade candidates; these downgrade and upgrade candidates are then compiled into a candidate list. Based on the real-time load of the storage system and the storage tier status of the storage manager, the migration costs and benefits of the candidate list are estimated, the migration targets are confirmed, and a migration plan is generated. Monitor the storage layer status in real time to generate intelligent scheduling solutions.

[0010] In one embodiment of the present invention, predictive analysis is performed on key metrics of the storage manager in the current cycle, and adaptive optimization is performed on the preset fusion model and policy scheduler in the next cycle, including: Collect key metrics of the storage manager within the current period; Conduct predictive analysis on key indicators to confirm the accuracy of the forecasts; If the short-term prediction in the prediction accuracy is inaccurate, adjust the decay factor in the preset fusion model for the next period; if the long-term trend prediction deviation in the prediction accuracy is greater than the preset deviation threshold, update the learning parameters of the preset fusion model for the next period. By using a reinforcement learning framework, the hierarchical decisions in the policy scheduler are treated as actions, and the storage system performance metrics are used as rewards to adjust the preset degradation threshold and preset upgrade threshold in the policy scheduler for the next cycle.

[0011] In one embodiment of the present invention, the key indicators include: Hit rates at each level of storage, improvement / deterioration of application I / O latency, accuracy of migration operations, and resource overhead.

[0012] In a second aspect, the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the data hierarchical processing method based on the storage manager provided by the present invention.

[0013] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of a data hierarchical processing method based on a storage manager provided in the embodiments of the present invention.

[0014] The beneficial effects of this invention are: The solution provided by this invention, through a triple mechanism of resource pool isolation, adaptive rate control, and intelligent timing scheduling, ensures that data migration operations will not cause perceptible performance interference to critical front-end services under any load, meeting the stringent requirements of high-performance applications for stability and low latency. The introduced feedback optimization closed loop enables the system to learn autonomously from historical decision results, dynamically adjusting prediction models, decision thresholds, and scheduling strategies. This allows the system to continuously adapt to complex and ever-changing business environments, achieving long-term optimal operation and eliminating tedious manual tuning. Furthermore, by integrating long-term and short-term time-series analysis with correlation analysis, this invention enables the system to predict future data value, proactively and in advance optimizing data layout. This solves the inherent decision-making lag problem of traditional methods, fundamentally improving the hit rate of the high-speed layer. Supporting sub-file level value assessment and migration, it achieves "data migration by popularity" rather than "transferring the entire data container," greatly reducing invalid I / O, improving the effective carrying capacity of the high-speed layer space, and reducing storage media wear. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the steps of a data hierarchical processing method based on a storage manager, provided in an embodiment of the present invention. Figure 2 A simulation heatmap of a data hierarchical processing method based on a storage manager provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0017] This invention provides a data hierarchical processing method, an electronic device, and a storage medium based on a storage manager.

[0018] Below, we will first introduce a data hierarchical processing method based on a storage manager provided by an embodiment of the present invention.

[0019] The present invention provides a data hierarchical processing method based on a storage manager, such as... Figure 1 As shown, it may include the following steps: S1, within each cycle, continuously monitors all I / O requests from the storage manager and extracts multi-dimensional dynamic feature vectors of data units in the monitoring storage manager, which may include: Continuously monitor all I / O requests from the storage manager, capture metadata of all read and write requests, and generate access event streams; Real-time analysis of access event streams extracts multi-dimensional dynamic feature vectors for each data unit.

[0020] Specifically, the storage manager captures metadata about all read and write requests with low overhead through storage drivers, virtualization layers, or network interception, generating an access event stream. Each event includes at least: data unit identifier (globally unique ID + LBA offset range), operation type (read / write), timestamp, request size, request source (application / virtual machine ID), and response latency.

[0021] Multidimensional dynamic feature vectors include: Recent popularity features, temporal pattern features, spatial correlation features, and business semantic features.

[0022] Recent popularity characteristics: Based on a sliding time window, the frequency of visits and the intensity of recent visits are calculated as recent popularity characteristics.

[0023] Access strength = ; in, This represents the exponential decay factor.

[0024] Time-series pattern features: Analyze access sequences over longer periods (such as 24 hours or 7 days) and extract periodic (such as daily peaks), trend (increase / decrease in access volume) and explosive features as time-series pattern features.

[0025] Spatial association features: Identify patterns such as sequential access and stride access, and construct an association graph between data units as spatial association features.

[0026] Business semantic features: Based on the request source and I / O mode (random / sequential, size), associate business types (such as database logs, video streams) and assign semantic tags to data units as business semantic features.

[0027] Understandably, step S1 is the data foundation for subsequent steps, and its core is to extract structured information with predictive value from raw I / O.

[0028] S2, based on multi-dimensional dynamic feature vectors, uses a pre-defined fusion model to calculate the hierarchical value of each data unit, thus obtaining a hierarchical value score for each data unit. Using a pre-defined fusion model, short-term value components, long-term value components, related value components, and cost value components are obtained based on multi-dimensional dynamic feature vectors. Based on short-term value components, long-term value components, related value components, and cost value components, a hierarchical value score for each data unit is calculated using a comprehensive dynamic weighting. .

[0029] The tiered value score for each data unit represents the expected benefits (such as reduced latency and increased throughput) that the data could gain from being stored in a higher-performance storage layer.

[0030] Specifically, the short-term value component Vs is mainly determined by recent popularity characteristics, reflecting the demand for immediate access.

[0031] The long-term value component Vl is confirmed by using time-series prediction models (such as LSTM and Prophet) to predict the access probability of the data unit in the next period (such as the next hour) based on time-series pattern characteristics.

[0032] The associated value component Vc is determined based on spatial association features. When a "parent" unit is predicted to be of high value, its strongly associated "child" units will also receive an increased Vc, which is used to achieve predictive prefetching.

[0033] The cost-value component (Vo) can be determined by combining business semantic features and system strategies. Data from high-priority businesses has higher fundamental value; at the same time, considering data size and migration costs, the Vo of excessively large but low-frequency data will decrease.

[0034] The expression for the graded value score is as follows: ; Indicates short-term dynamic weights. Indicates long-term dynamic weights. Indicates the dynamic weight of association. This indicates the dynamic weight of costs.

[0035] Understandably, , , and It is not static, but rather time-related. t The function is as follows: During peak business hours, the weight of Vs is increased for rapid response; during off-peak hours, the weights of Vl and Vc are increased for more proactive, prediction-based data layout optimization. This dynamic weighting mechanism ensures the adaptability of the strategy to business patterns. Step S2 transforms the static features extracted in the previous step into dynamic predictions of future data value, providing a direct basis for hierarchical decision-making.

[0036] S3, utilizing a policy scheduler, generates migration plans and intelligent scheduling schemes based on tiered value scores, real-time load of the storage system, and storage tier status of the storage manager. These schemes may include: Using a policy scheduler, performance layers in data units with tiered value scores below a preset downgrade threshold are marked as downgrade candidates, and capacity layers in data units with tiered value scores above a preset upgrade threshold are marked as upgrade candidates; these downgrade and upgrade candidates are then compiled into a candidate list. Based on the real-time load of the storage system and the storage tier status of the storage manager, the migration costs and benefits of the candidate list are estimated, the migration targets are confirmed, and a migration plan is generated. Monitor the storage layer status in real time to generate intelligent scheduling solutions.

[0037] Specifically, the policy scheduler runs the decision-making process periodically.

[0038] The policy scheduler receives three types of input: the hierarchical value score of all data units, the real-time load of the storage system (CPU utilization, total I / O bandwidth, foreground I / O latency), and the storage tier status of the storage manager (remaining capacity, health, current load).

[0039] Generate migration plan: The data in each tier is sorted according to the hierarchical value score. For the performance tier, Below the preset degradation threshold θ demote Data is marked as a degradation candidate; for the capacity layer, Higher than the preset upgrade threshold θ promote The data is marked as upgrade candidates; the preset downgrade threshold θ demote and preset upgrade threshold θ promote It can be dynamically adjusted; when the performance layer has insufficient free space, the preset degradation threshold θ is automatically lowered. demote To free up more space, demotion and upgrade candidates will be included in the candidate list.

[0040] For the candidate list, estimate the migration costs (required bandwidth and time) and benefits (expected reduction in access latency after migration), and retain only candidates with a benefit-cost ratio higher than the set value or those that are key candidates (such as...). Tasks with extremely high latency or limited space in their original location are selected as migration targets to generate a migration plan (source data unit ID, target level, priority).

[0041] Intelligent scheduling solutions may include: Resource Quota and Isolation: Allocate an independent, elastically scalable resource pool (such as bandwidth limits and I / O queues) to the migration engine and physically or logically isolate it from front-end service I / O. The capacity of this resource pool can be dynamically adjusted according to subsequent adaptive rate control strategies (e.g., when the system is idle, the license pool capacity = 50% of the total bandwidth; when the system is busy, it automatically drops to 5%). Subsequent step S4 must request and obtain a license token from this license pool before it can perform the actual data transfer. Each token represents a fixed bandwidth quota and I / O queue resource.

[0042] Adaptive rate control: Real-time monitoring of foreground application I / O latency. If latency exceeds a safety threshold, the migration engine's bandwidth quota is reduced proportionally to ensure absolute priority for foreground performance.

[0043] Timing optimization: Based on the system idle periods identified by historical load patterns, the migration quota is automatically increased during these periods to accelerate the completion of non-urgent large-scale data degradation or reorganization tasks.

[0044] The policy scheduler can execute migration plans sequentially based on a preset priority queue order and the number of currently available license tokens.

[0045] Understandably, step S3 is the decision-making center of the data hierarchical processing method based on the storage manager proposed in this application. It connects the data value prediction obtained in step S2 with the physical resource status, and solves the problems of when, what data, and what migration to perform.

[0046] S4, based on migration plans and intelligent scheduling schemes, performs atomic and non-disruptive migration of data units to be migrated in order to maintain data consistency.

[0047] Specifically, the migration process may include: Preparation phase: Based on the migration plan and intelligent scheduling scheme, lock the metadata (such as mapping items) of the data units to be migrated to prevent concurrent writes. Start data replication to the target layer.

[0048] Redirection and Switching: A write-on-demand redirection technique is used for migration. During migration, new write operations on the data are redirected to a new location in the target layer, and a redirection pointer is recorded at the original location. Read operations can choose between source and target based on an optimization strategy. During migration, the data transfer rate is limited using a token bucket algorithm based on the obtained permission tokens.

[0049] Commit Phase: After data replication is complete, the global metadata mapping table is atomically updated, pointing the logical address to the new physical location in the target layer. This operation is an atomic transaction, ensuring data consistency from the client's perspective.

[0050] Cleanup phase: Asynchronously release old data blocks on the source storage layer.

[0051] The entire migration process is transparent to the application; the application does not need to be aware of the data location change, and access will not be interrupted.

[0052] S5 performs predictive analysis on key metrics of the storage manager in the current cycle and adaptively optimizes the preset fusion model and policy scheduler for the next cycle, which may include: Collect key metrics of the storage manager within the current period; Conduct predictive analysis on key indicators to confirm the accuracy of the forecasts; If the short-term prediction in the prediction accuracy is inaccurate, adjust the decay factor in the preset fusion model for the next period; if the long-term trend prediction deviation in the prediction accuracy is greater than the preset deviation threshold, update the learning parameters of the preset fusion model for the next period. By using a reinforcement learning framework, the hierarchical decisions in the policy scheduler are treated as actions, and the storage system performance metrics are used as rewards to adjust the preset degradation threshold and preset upgrade threshold in the policy scheduler for the next cycle.

[0053] Key metrics may include: Hit rates at each level of storage, improvement / deterioration of application I / O latency, accuracy of migration operations, and resource overhead.

[0054] Specifically, the data grading processing method provided in this embodiment of the invention continuously collects key indicators such as the hit rate of each level of storage, the improvement / deterioration of application I / O latency, the accuracy of migration operations (e.g., whether the upgraded data is frequently accessed subsequently), and resource overhead.

[0055] The difference between the predicted access probability when the data is upgraded and the actual access frequency thereafter is compared through predictive analysis.

[0056] Adaptive optimization of model parameters and policies can include: Model parameter tuning: If short-term predictions are found to be inaccurate, adjust the exponential decay factor in the recent popularity characteristics. If the long-term trend prediction has a large deviation, the online learning function of the time series prediction model is used to update the model.

[0057] Decision threshold tuning: Utilizing a reinforcement learning framework. The hierarchical decision-making process in the policy scheduler is treated as an action, with storage system performance metrics (such as overall average latency) as the reward. Through continuous trial and error, θ is fine-tuned. promote θ demote With equal thresholds, learn the optimal strategy that maximizes long-term rewards.

[0058] Policy library update: For specific business patterns identified (such as weekly batch processing), specific hierarchical policy templates can be generated or adapted, and automatically enabled when the same pattern occurs.

[0059] Understandably, step S5 constitutes the learning loop of the data hierarchical processing method. It evaluates the effectiveness of the decision-making and execution cycles formed by the first three steps and feeds them back to the system to drive its continuous improvement, which is the key to achieving adaptability and intelligence.

[0060] The simulation heatmap of the data hierarchical processing method provided in this embodiment of the invention is as follows: Figure 2 As shown in the figure, the proportion of hot data areas and high-performance storage resources is well matched, which conforms to the Pareto principle and ensures timely data migration.

[0061] The data tiered processing method provided in this invention, through a triple mechanism of resource pool isolation, adaptive rate control, and intelligent timing scheduling, ensures that data migration operations will not cause perceptible performance interference to critical front-end services under any load, meeting the stringent requirements of high-performance applications for stability and low latency. The introduced feedback optimization closed loop enables the system to learn autonomously from historical decision results, dynamically adjusting prediction models, decision thresholds, and scheduling strategies. This allows the system to continuously adapt to complex and ever-changing business environments, achieving long-term optimal operation and eliminating tedious manual tuning. By integrating long-term and short-term time-series analysis with correlation analysis, the data tiered processing method enables the system to predict future data value, proactively and in advance optimizing data layout. This solves the inherent decision-making lag problem of traditional methods, fundamentally improving the hit rate of the high-speed layer. Supporting sub-file level value assessment and migration, it achieves "data migration by popularity" rather than "transferring the entire data container," greatly reducing invalid I / O, improving the effective carrying capacity of the high-speed layer space, and reducing storage media wear.

[0062] For ease of understanding, the present invention will be described in detail below with reference to a preferred embodiment in a hyperconverged infrastructure environment. This embodiment is for implementation purposes only and should not be construed as limiting the scope of protection of the present invention.

[0063] Implementation Environment A hyperconverged cluster consisting of 3 nodes, each equipped with an NVMe SSD (as a performance tier) and a high-capacity SATA HDD (as a capacity tier). All storage is virtualized into a unified distributed data store.

[0064] Implementation steps deploy: The data hierarchical processing method provided in this embodiment of the invention is deployed on each node of the storage cluster in the form of an independent virtual machine or a containerized microservice, and a master manager is selected. The manager is started and the default policy is loaded.

[0065] Initial monitoring and learning: The manager begins monitoring all virtual machine I / O to the virtual disk. Initially, a value rating is assigned. Primarily relying on recent trending features (Vs). After running for several hours, the analysis engine identified the following patterns: Virtual Machine (VM) DB The log file area experiences strong periodic access peaks every day from 9:00 to 10:00 AM.

[0066] Virtual Machine (VM) Web For static resource files, once the first part is read sequentially, there is a greater than 70% probability that the subsequent parts will be read within 1 second.

[0067] Foresightful decision-making and execution: At 8:50 AM the following day, the master manager predicted the VM based on the periodic pattern. DB The log section is about to become a hot topic. Although its current Vs is not high, its long-term value Vl has been adjusted upwards, overall... The value exceeds the preset upgrade threshold. The decision-maker generates an upgrade task.

[0068] At the same time, VM was detected. Web The process begins by sequentially reading the beginning of a large file. Based on the correlation model Vc, it immediately predicts that subsequent data blocks will be accessed and generates a prefetch upgrade task.

[0069] At this time, the overall system load is low, and the scheduler executes these upgrade tasks with a higher bandwidth quota (such as 30% of the total bandwidth) to complete the warm-up of critical data before 9:00.

[0070] Adaptive disturbance control: At 9:00 AM, the peak business period arrived, and front-end I / O latency increased. Monitoring showed that latency was approaching the threshold, and the scheduler immediately and automatically reduced the migration engine bandwidth quota to 5%.

[0071] In the afternoon, VM_DB performed batch data archiving, generating a large amount of cold data. The V value of this data decreased over time. The decision-maker generated a degradation task, but it was postponed by the scheduler due to its low priority.

[0072] Off-peak optimization and feedback: At night, the system detects that the load has entered an idle period. The scheduler automatically increases the migration bandwidth quota to 50%, accelerating the execution of degradation tasks that were backlogged during the day, and efficiently moving cold data out of the performance layer.

[0073] The following day, the feedback module analysis revealed that, for VM Web The associated prefetch accuracy reached 85%, effectively reducing user access latency. The model automatically enhanced the Vc weight for this type of sequential access pattern. Simultaneously, upon discovering inaccurate predictions for a small random file library, the system automatically reduced the long-term prediction weight Vl for that region of data, preventing subsequent erroneous upgrades.

[0074] It is understood that the data hierarchical processing method provided in the embodiments of the present invention has the following advantages compared with the prior art: A qualitative leap from reactive to predictive: By integrating long-term and short-term time series analysis with correlation analysis, the system has the ability to predict the future value of data, and can proactively and in advance optimize the data layout, solving the inherent problem of decision lag in traditional methods and fundamentally improving the hit rate of high-speed layers.

[0075] Refined decision-making and efficient resource utilization: It supports value assessment and migration at the sub-file level, realizes data transfer based on popularity rather than transferring the entire data container, greatly reduces invalid I / O, improves the effective carrying capacity of the high-speed layer space, and reduces storage media wear.

[0076] Reliability of performance isolation and business assurance: Through resource pool isolation, adaptive rate control and intelligent timing scheduling, the system ensures that data migration operations will not cause perceptible performance interference to front-end critical businesses under any load conditions, meeting the stringent requirements of high-performance applications for stability and low latency.

[0077] The continuous optimization capability of a self-evolving system: The introduced feedback optimization closed loop enables the system to learn autonomously from historical decision results, dynamically adjust prediction models, decision thresholds and scheduling strategies, so that the system can continuously adapt to complex and ever-changing business environments, achieve long-term optimal operation, and eliminate tedious manual tuning.

[0078] Potential for global resource collaborative optimization: The storage manager architecture designed based on this invention inherently supports distributed scaling. Multiple managers can work collaboratively, sharing a global heat map, achieving data balancing and hotspot elimination across nodes and racks, and realizing optimal resource allocation at the cluster level.

[0079] Secondly, embodiments of the present invention also provide an electronic device, such as... Figure 3 As shown, it includes a processor 001, a communication interface 002, a memory 003, and a communication bus 004, wherein the processor 001, the communication interface 002, and the memory 003 communicate with each other through the communication bus 004. The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements any of the data hierarchical processing methods based on the storage manager provided in the first aspect of the present invention.

[0080] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0081] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0082] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0083] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0084] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0085] Thirdly, corresponding to the data hierarchical processing method based on a storage manager provided in the first aspect, this embodiment of the invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the data hierarchical processing methods based on a storage manager provided in the first aspect of this invention.

[0086] For the embodiments of the device / electronic device / storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments.

[0087] It should be noted that the electronic device and storage medium in the embodiments of the present invention are respectively electronic devices and storage media applying the above method. Therefore, all embodiments of the above method are applicable to the electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0088] It should be noted that, in the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0089] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data hierarchical processing method based on a storage manager, characterized in that, include: Within each cycle, continuously monitor all I / O requests of the storage manager and extract multi-dimensional dynamic feature vectors of data units in the monitoring storage manager; Based on multidimensional dynamic feature vectors, the hierarchical value of each data unit is calculated using a pre-defined fusion model, and the hierarchical value score of each data unit is obtained. Using a policy scheduler, migration plans and intelligent scheduling schemes are generated based on hierarchical value scores, real-time load of the storage system, and storage layer status of the storage manager. Based on migration plans and intelligent scheduling schemes, the data of the data units to be migrated is migrated atomically and without interference to maintain data consistency; Perform predictive analysis on key metrics of the storage manager in the current cycle, and adaptively optimize the preset fusion model and policy scheduler for the next cycle.

2. The data hierarchical processing method based on a storage manager according to claim 1, characterized in that, The continuous monitoring of all I / O of the storage manager extracts multi-dimensional dynamic feature vectors of data units in the monitored storage manager, including: Continuously monitor all I / O requests from the storage manager, capture metadata of all read and write requests, and generate access event streams; Real-time analysis of access event streams extracts multi-dimensional dynamic feature vectors for each data unit.

3. The data hierarchical processing method based on a storage manager according to claim 1, characterized in that, The multidimensional dynamic feature vector includes: Recent popularity features, temporal pattern features, spatial correlation features, and business semantic features.

4. The data hierarchical processing method based on a storage manager according to claim 1, characterized in that, The step of calculating the hierarchical value of each data unit based on multidimensional dynamic feature vectors using a preset fusion model to obtain a hierarchical value score for each data unit includes: Using a pre-defined fusion model, short-term value components, long-term value components, related value components, and cost value components are obtained based on multi-dimensional dynamic feature vectors. The hierarchical value score of each data unit is calculated by comprehensively considering the short-term value component, long-term value component, related value component, and cost value component, with dynamic weights.

5. The data hierarchical processing method based on a storage manager according to claim 1, characterized in that, The policy scheduler, based on hierarchical value scoring, real-time load of the storage system, and storage layer status of the storage manager, generates migration plans and intelligent scheduling schemes, including: Using a policy scheduler, performance layers in data units with tiered value scores below a preset downgrade threshold are marked as downgrade candidates, and capacity layers in data units with tiered value scores above a preset upgrade threshold are marked as upgrade candidates; these downgrade and upgrade candidates are then compiled into a candidate list. Based on the real-time load of the storage system and the storage tier status of the storage manager, the migration costs and benefits of the candidate list are estimated, the migration targets are confirmed, and a migration plan is generated. Monitor the storage layer status in real time to generate intelligent scheduling solutions.

6. The data hierarchical processing method based on a storage manager according to claim 5, characterized in that, The process of predictively analyzing key metrics of the storage manager in the current cycle and adaptively optimizing the preset fusion model and policy scheduler for the next cycle includes: Collect key metrics of the storage manager within the current period; Conduct predictive analysis on key indicators to confirm the accuracy of the forecasts; If the short-term prediction in the prediction accuracy is inaccurate, adjust the decay factor in the preset fusion model for the next period; if the long-term trend prediction deviation in the prediction accuracy is greater than the preset deviation threshold, update the learning parameters of the preset fusion model for the next period. By using a reinforcement learning framework, the hierarchical decisions in the policy scheduler are treated as actions, and the storage system performance metrics are used as rewards to adjust the preset degradation threshold and preset upgrade threshold in the policy scheduler for the next cycle.

7. The data hierarchical processing method based on a storage manager according to claim 6, characterized in that, The key indicators include: Hit rates at each level of storage, improvement / deterioration of application I / O latency, accuracy of migration operations, and resource overhead.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the steps of the data hierarchical processing method based on a storage manager as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the data hierarchical processing method based on a storage manager as described in any one of claims 1-7.

Images