Cloud resource-based data tiered storage migration service system

Abstract

The application discloses a cloud resource-based data hierarchical storage migration service system and relates to the technical field of data migration.The system comprises a cost decision unit, a batch planning unit and a migration execution unit.The cost decision unit is used for establishing a cost-optimized migration decision mechanism for data blocks in different storage levels according to cloud resource platform charging rules and price data, and determining a migration timing scheduling table.The batch planning unit is used for constructing a migration demand distribution matrix and analyzing a resource constraint influence coefficient based on the migration timing scheduling table and a real-time state of the cloud resource platform, and generating a migration planning batch.The migration execution unit is used for executing migration operations of data blocks in different storage levels according to the migration timing scheduling table and the migration planning batch.The application can dynamically optimize migration decisions according to the elastic charging characteristics of cloud resources, and solves the problems that the prior art lacks in-depth utilization of the elastic charging characteristics of cloud resources and fails to realize intelligent planning of migration timing and batches.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data migration, in particular to a data hierarchical storage migration service system based on cloud resources. BACKGROUND

[0002] With the rapid development of cloud computing technology, enterprise data storage needs are showing a trend of diversification, and data hierarchical storage technology has emerged as the times require. The data hierarchical storage migration service system based on cloud resources has become a key requirement for enterprise data management, which can dynamically allocate data of different temperatures to storage levels that match performance and cost, and achieve optimal configuration of storage resources. However, when enterprise data is scattered in multi-cloud and hybrid cloud environments, how to perform cost-optimized data migration according to the elastic billing characteristics of cloud resources has become a technical problem that needs to be solved urgently, and enterprises need a system that can identify the best migration opportunity and optimize the migration batch to achieve cross-platform, efficient and low-cost data management.

[0003] At present, there are many data storage and migration solutions on the market, mainly including data migration services provided by cloud service providers and third-party data management platforms. These tools provide basic data transfer functions and simple hierarchical storage strategies, such as automatically adjusting data storage levels according to access frequency; at the same time, some hybrid cloud data management platforms support cross-cloud data operations, providing basic support for enterprise data governance. These technical solutions have alleviated the pressure of enterprise data management to a certain extent, provided technical support for enterprise data governance, and promoted the process of migrating enterprise IT infrastructure to the cloud.

[0004] Although the existing technology has made certain progress in the field of data migration and hierarchical storage, there are still many deficiencies. First of all, most solutions only focus on data management within a single cloud platform, cannot adapt to the elastic billing mode of cloud storage, and cannot dynamically adjust the migration decision according to real-time pricing strategies; secondly, the existing system lacks intelligent judgment of migration timing and cannot identify the best migration window within the billing period, resulting in increased migration costs; thirdly, traditional tools do not adequately consider resource constraints in cloud environments and cannot dynamically adjust the migration batch according to network bandwidth and storage performance, nor do they lack comprehensive consideration of multi-dimensional costs of storage, access and migration.

[0005] In view of the problems in the related art, no effective solutions have been proposed so far. SUMMARY

[0006] In view of the problems in the prior art, the application provides a data hierarchical storage migration service system based on cloud resources, which has the advantages of dynamically optimizing migration decisions according to the characteristics of cloud resources, and further solves the problems that the existing technology lacks in-depth utilization of the elastic billing characteristics of cloud resources, cannot optimize migration decisions according to price changes, and cannot realize intelligent planning of migration timing and batches.

[0007] To this end, the application adopts the following specific technical solutions:

[0008] The data hierarchical storage migration service system based on cloud resources comprises:

[0009] A cost decision unit is configured to establish a cost-optimized migration decision mechanism for data blocks in different storage levels according to the billing rules and price data of the cloud resource platform, and determine a migration timing scheduling table.

[0010] A batch planning unit is configured to construct a migration demand distribution matrix and analyze resource constraint influence coefficients based on the migration timing scheduling table and the real-time state of the cloud resource platform, and generate a migration planning batch.

[0011] A migration execution unit is configured to execute migration operations of data blocks in different storage levels according to the migration timing scheduling table and the migration planning batch.

[0012] The cost decision unit comprises:

[0013] A cost modeling module is configured to establish a multi-dimensional cost calculation model by determining hierarchical storage costs, data access costs and migration operation costs based on the billing rules and price data of the cloud resource platform.

[0014] An index evaluation module is configured to analyze cost influence factors of data migration using the multi-dimensional cost calculation model, and generate elastic billing evaluation indexes.

[0015] A decision optimization module is configured to construct a cost-optimized migration decision function according to the elastic billing evaluation indexes, and determine the migration timing scheduling table.

[0016] Further, the cost decision unit is connected to the batch planning unit and the migration execution unit; and the cost modeling module comprises:

[0017] A data acquisition module is configured to acquire hierarchical storage billing data, data access billing data and migration operation billing data of the cloud resource platform through an API interface, and establish a multi-dimensional billing data set.

[0018] A cost analysis module is configured to analyze hierarchical storage costs, data access costs and migration operation costs based on the multi-dimensional billing data set, in combination with different storage level price coefficients and billing period information of the cloud resource platform.

[0019] a model construction module configured to establish a multi-dimensional cost calculation model according to the tiered storage cost, the data access cost and the migration operation cost.

[0020] Further, the analysis of the tiered storage cost, the data access cost and the migration operation cost comprises: based on tiered storage billing data, extracting the unit capacity price, the minimum storage duration requirement and the early deletion penalty fee of different storage levels, combining the actual occupied space of the data block and the expected retention period, obtaining the tiered storage cost; according to the read-write operation unit price, the data retrieval fee and the bandwidth rate in the data access billing data, combining the historical access mode and the access frequency statistical information, calculating the data access cost; using the data transmission fee, the computing resource occupation fee and the cross-region additional fee in the migration operation billing data, analyzing the billing low valley period and peak period of different migration periods, determining the migration operation cost.

[0021] Further, the index evaluation module comprises:

[0022] a cost estimation module configured to determine the current storage cost, the access cost and the operation cost of the potential migration path of each data block distributed in different storage levels based on the multi-dimensional cost calculation model and the historical storage record of the cloud resource platform;

[0023] a factor extraction module configured to calculate the cost change rate, the resource utilization efficiency and the billing period coefficient under different migration strategies according to the current storage cost, the access cost and the operation cost of the potential migration path of each data block, and generate key cost influence factors;

[0024] an index construction module configured to establish an elastic billing evaluation index including the storage level sensitivity, the access frequency threshold and the migration timing critical value based on the key cost influence factors.

[0025] Further, the decision optimization module comprises:

[0026] a function establishment module configured to establish a migration target function based on the elastic billing evaluation index and with the minimization of migration cost as the target, combining the access performance constraint condition;

[0027] a timing identification module configured to calculate the cost-benefit score of performing migration operation at different time points by using the migration target function, and identify the migration time window with the lowest cost;

[0028] a scheduling generation module configured to determine the target migration timing and the priority ranking of each data block according to the cost-benefit score and the migration urgency, and establish a migration scheduling table.

[0029] Further, the batch planning unit comprises:

[0030] The demand analysis module is configured to analyze time distribution characteristics of target migration time of each data block according to a migration time scheduling table, and construct a migration demand distribution matrix in combination with an elastic billing evaluation index;

[0031] The constraint evaluation module is configured to analyze a resource constraint influence coefficient according to a real-time bandwidth state and a storage level load condition of the cloud resource platform, and dynamically adjust migration parameters based on a preset threshold value;

[0032] The batch generation module is configured to generate a migration planning batch based on the migration demand distribution matrix and the resource constraint influence coefficient.

[0033] Further, the demand analysis module comprises:

[0034] The information extraction module is configured to extract target migration time and priority order information of each data block based on the migration time scheduling table;

[0035] The feature recognition module is configured to analyze time distribution characteristics of target migration time of each data block, and identify migration demand density and resource competition degree in a billing period;

[0036] The matrix construction module is configured to construct the migration demand distribution matrix in combination with the migration demand density, the resource competition degree, and storage level sensitivity and access frequency threshold values in the elastic billing evaluation index.

[0037] Further, the resource constraint influence coefficient is analyzed according to a real-time bandwidth state and a storage level load condition of the cloud resource platform, and the migration parameters are dynamically adjusted based on a preset threshold value, which comprises: calculating the resource constraint influence coefficient; if the network bandwidth availability of the cloud resource platform is lower than a bandwidth safety threshold value, then reducing the migration data volume in a unit time and correspondingly increasing the migration batch quantity according to a bandwidth adjustment ratio; if the IOPS usage rate of a target storage level exceeds a load upper threshold value, then adjusting the migration execution time to a time period in which the IOPS usage rate of the storage level is lower than a load suitable threshold value; if the access operation quantity of a source storage level increases at a rate exceeding an access fluctuation threshold value in a monitoring period, then delaying non-critical migration tasks to the next billing period and recalculating the resource constraint influence coefficient.

[0038] Further, the resource constraint influence coefficient is calculated, which comprises: acquiring network bandwidth usage, storage level IOPS load level and access operation frequency of the cloud resource platform in real time through an API interface, and solving the resource constraint influence coefficient; an expression of the resource constraint influence coefficient is:

[0039] ;

[0040] In the formula, C is the resource constraint influence coefficient; IC BW r ​a current network bandwidth availability rate; BW max a maximum network bandwidth availability rate; IOPS c a current IOPS usage rate of the target storage tier; IOPS max a maximum IOPS threshold of the target storage tier; AO r a growth rate of the number of access operations of the source storage tier; AO max a maximum allowed value of the access operation growth rate; α 、 β 、 γ are weight coefficients of the network bandwidth factor, the IOPS load factor and the access operation factor, respectively.

[0041] Further, the batch generation module comprises:

[0042] a scheme formulation module configured to determine a migration priority according to the migration demand distribution matrix, and establish a batch allocation scheme for each data block in combination with the resource constraint influence coefficient;

[0043] a task merging module configured to merge migration tasks in the same billing period in the batch allocation scheme based on billing period information of the cloud resource platform, so as to reduce migration operation cost;

[0044] a planning output module configured to generate a migration planning batch comprising a batch sequence, a predicted execution time and a resource demand amount, and synchronize the migration planning batch with a migration timing schedule in time.

[0045] The present application has the following advantages:

[0046] (1) The present application establishes a multi-dimensional cost calculation model based on the elastic billing characteristics of the cloud resource platform, and combines the storage tier sensitivity, the access frequency threshold and the migration timing critical value and other elastic billing evaluation indexes, so that the system can accurately identify the cost-optimal migration window in the billing period, and dynamically adjust the migration batch according to the resource constraint influence coefficient, thereby effectively reducing the total data storage cost of the enterprise in the multi-cloud environment. According to tests, the cost can be saved by 15%-30% in typical application scenarios, and the cost optimization effect will be more significant as the data size grows and the cloud resource price fluctuates.

[0047] (2) Based on the double constraint mechanism of migration demand distribution matrix and resource constraint influence coefficient, the system can intelligently identify the migration demand density and resource competition degree in the billing period, optimize and integrate the migration tasks in the same billing period through the task integration module, and automatically adjust the migration parameters when the key resource indicators such as network bandwidth, storage IOPS and access operation approach the threshold, which not only ensures the business continuity and system stability, but also maximizes the resource utilization rate through intelligent planning and dynamic adjustment of migration batches, significantly improves the data migration efficiency and reduces the management complexity. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0049] Figure 1 is a principle block diagram of a data hierarchical storage migration service system based on cloud resources according to an embodiment of the present application;

[0050] Figure 2 is a specific implementation diagram of a cost decision unit in a data hierarchical storage migration service system based on cloud resources according to an embodiment of the present application;

[0051] Figure 3 is a specific implementation diagram of a constraint evaluation module in a data hierarchical storage migration service system based on cloud resources according to an embodiment of the present application;

[0052] Figure 4 is a specific implementation diagram of a batch generation module in a data hierarchical storage migration service system based on cloud resources according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] To further illustrate the embodiments, the present application provides drawings, which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. With reference to these contents, those skilled in the art should understand other possible embodiments and advantages of the present application. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0054] According to an embodiment of the present application, a data hierarchical storage migration service system based on cloud resources is provided.

[0055] The present application will be further described in conjunction with the drawings and specific embodiments, such as Figure 1As shown, according to one embodiment of the present application, a cloud resource-based data tiered storage migration service system is provided, which comprises:

[0056] a cost decision unit 1 configured to establish a cost-optimized migration decision mechanism for data blocks in different storage tiers according to cloud resource platform billing rules and price data, and determine a migration timing schedule;

[0057] a batch planning unit 2 configured to construct a migration demand distribution matrix and analyze resource constraint influence coefficients based on the migration timing schedule and real-time state of the cloud resource platform, and generate a migration planning batch;

[0058] a migration execution unit 3 configured to execute migration operations for data blocks in different storage tiers according to the migration timing schedule and the migration planning batch;

[0059] Specifically, the cost decision unit 1 is connected to the batch planning unit 2 and the migration execution unit 3.

[0060] In one embodiment, as shown, Figure 2 the cost decision unit 1 comprises:

[0061] a cost modeling module configured to establish a multi-dimensional cost calculation model by determining tiered storage costs, data access costs, and migration operation costs based on cloud resource platform billing rules and price data;

[0062] an index evaluation module configured to analyze cost influence factors of data migration using the multi-dimensional cost calculation model, and generate elastic billing evaluation indexes;

[0063] a decision optimization module configured to construct a cost-optimized migration decision function according to the elastic billing evaluation indexes, and determine a migration timing schedule.

[0064] In one embodiment, the cost modeling module comprises:

[0065] a data collection module configured to obtain tiered storage billing data, data access billing data, and migration operation billing data of the cloud resource platform through an API interface, and establish a multi-dimensional billing data set;

[0066] a cost analysis module configured to analyze tiered storage costs, data access costs, and migration operation costs based on the multi-dimensional billing data set, in combination with different storage tier price coefficients and billing period information of the cloud resource platform;

[0067] a model construction module configured to establish a multi-dimensional cost calculation model according to the tiered storage costs, data access costs, and migration operation costs.

[0068] In one embodiment, analyzing the hierarchical storage cost, data access cost and migration operation cost comprises: based on the hierarchical storage billing data, extracting the unit capacity price, minimum storage time requirement and early deletion penalty fee of different storage levels, combining the actual occupied space of the data block and the expected retention period, obtaining the hierarchical storage cost; according to the read-write operation unit price, data retrieval fee and bandwidth rate in the data access billing data, combining the historical access mode and access frequency statistical information, the data access cost is calculated; using the data transmission fee, computing resource occupation fee and cross-region additional fee in the migration operation billing data, analyzing the billing low period and peak period of different migration periods, and determining the migration operation cost.

[0069] Specifically, the present application realizes the accurate calculation and prediction of the multi-level storage migration cost in the cloud environment through the cost modeling module. In the actual implementation process, the module first connects the open API interface of each major cloud service provider (such as Ali Cloud, Tencent Cloud, AWS, etc.) through the data acquisition module, and regularly (for example, every 6 hours) obtains the latest billing rules and price parameters. The system will convert these raw data into a standardized JSON format, and build a multi-dimensional billing data set containing three dimensions (storage, access, migration), which is convenient for subsequent analysis and processing.

[0070] Specifically, after obtaining the multi-dimensional billing data set, the cost analysis module will conduct in-depth cost analysis. Taking the hierarchical storage cost as an example, the system will extract the unit capacity price (such as 0.12 yuan per GB per month), the minimum storage period requirement (such as 30 days, 90 days, 180 days, etc.) and the penalty fee for early deletion for different storage levels (such as hot storage, warm storage, cold storage, archive storage, etc.). The system will then combine the actual occupied space size and the expected retention period of each data block in the actual business scenario of the enterprise to calculate the storage cost of each data block in different storage levels. For example, for a 100GB data block, if the expected retention period is 45 days, the system will calculate the cost difference between the hot storage layer (no minimum period limit) and the warm storage layer (assuming a minimum of 30 days), including possible early deletion penalty fees. Secondly, for the data access cost, the system calculates the access cost of each storage level based on the read-write operation price, request fee and data transmission rate in the cloud platform billing data, combined with the historical access records of the data block. Finally, the migration operation cost analysis mainly focuses on the data transmission fee, computing resource usage fee and possible cross-region additional fee. The system will identify the low period and peak period in the billing period of the cloud platform, providing a basis for subsequent migration batch planning. For example, the data transmission fee of a certain cloud service provider may be 30% lower from 2am to 6am than during working hours. The system will include this information in the multi-dimensional cost calculation model to optimize the migration timing selection.

[0071] Specifically, the model construction module establishes a multi-dimensional cost calculation model based on the analysis results of the above three types of costs. The model uses a matrix form to represent the cost relationship of different storage levels, different data blocks, and different time points, and the expression is:

[0072] C(b, l, t) = λ × S(b, l) + μ × A(b, l, t) + ω × M(b, l1, l2, t);

[0073] In the formula, C represents the total cost function, b represents the data block, l represents the storage level, t represents the migration execution time point, S represents the storage cost function, A represents the access cost function, M represents the migration cost function from the l1 level to the l2 level, λ, μ, ω are weight coefficients of the three types of costs, which can be dynamically adjusted according to the business focus of different data.

[0074] The multi-dimensional cost calculation model not only considers the static storage cost, but also includes the dynamic access cost and migration cost, so that the system can comprehensively evaluate the economy of different migration strategies in the subsequent decision-making process, and provide a cost-optimal solution for data migration.

[0075] In one embodiment, the index evaluation module comprises:

[0076] The cost estimation module is configured to determine the current storage cost, access cost, and operation cost of the potential migration path of each data block distributed in different storage levels based on the multi-dimensional cost calculation model and the historical storage records of the cloud resource platform.

[0077] The factor extraction module is configured to calculate the cost change rate, resource utilization efficiency, and billing period coefficient under different migration strategies according to the current storage cost, access cost, and operation cost of the potential migration path of each data block, and generate key cost influence factors.

[0078] The index construction module is configured to establish an elastic billing evaluation index including storage level sensitivity, access frequency threshold, and migration timing critical value based on the key cost influence factors.

[0079] Specifically, the index evaluation module of the present application realizes accurate evaluation and decision optimization of data migration in the cloud environment through a systematic cost analysis method. In actual implementation, the module first uses the cost estimation module to comprehensively evaluate enterprise data. Based on the multi-dimensional cost calculation model established as described above, the module combines the historical storage and access records of the cloud platform in the last 90 days to generate a detailed cost profile for each data block.

[0080] Specifically, the cost estimation module obtains actual cost data of each data block at different storage levels by calling a storage analysis API of the cloud resource platform. For example, for a 500 GB enterprise financial data set, the system calculates the monthly storage cost (e.g., 500 GB x 0.15 yuan / GB = 75 yuan) of the data set at the current hot storage level, the access cost (e.g., 10,000 requests per month x 0.01 yuan / request = 100 yuan), and the potential operation cost (e.g., 500 GB x 0.08 yuan / GB migration fee = 40 yuan) of migrating the data set to the cold storage level. The system simultaneously evaluates multiple possible migration paths (e.g., hot storage -> warm storage -> cold storage or hot storage -> archive storage, etc.) and calculates the complete cost structure of each path to form a multi-dimensional cost matrix.

[0081] Specifically, after obtaining detailed cost data, the factor extraction module first calculates the cost change rate under different migration strategies, that is, the percentage change of the total cost before and after migration, which is expressed as:

[0082] ΔC% = (C_after - C_before) / C_before x 100%;

[0083] In the formula, C_after represents the total cost after data migration, C_before represents the total cost before data migration, ΔC% represents the cost change percentage caused by the migration strategy, and a positive value represents an increase in cost and a negative value represents a cost saving.

[0084] Then, the resource utilization efficiency is evaluated, and the resource utility coefficient is calculated by analyzing the matching degree of the storage space utilization rate and the access frequency of the data block:

[0085] η = (S_utilization x A_frequency) / S_capacity;

[0086] In the formula, S_utilization represents the actual utilization rate of the storage space of the data block, A_frequency represents the matching degree of the data access frequency and the performance of the storage level (value range 0-1), S_capacity represents the total storage capacity allocated to the data block, η represents the resource utility coefficient, and a higher value indicates higher resource utilization efficiency.

[0087] Finally, the billing period coefficient is analyzed, which reflects the matching degree of the billing period of different storage levels and the data lifecycle:

[0088] τ = min(T_billing / T_lifecycle, 1) x f(T_remainder);

[0089] In the formula, T_billing represents the billing period of the storage hierarchy (such as month, quarter, year), T_lifecycle represents the expected life cycle of the data, T_remainder represents the remaining billing period time, f() is a smoothing function, τ represents the billing period coefficient, and the value closer to 1 indicates that the billing period and the data life cycle match better, which helps to reduce the storage cost. By comprehensively analyzing the three dimensions, the system extracts a set of key cost influence factors of data migration decision, providing a basis for subsequent evaluation index construction.

[0090] Specifically, the index construction module constructs an elastic billing multi-dimensional evaluation model based on the extracted key cost influence factors, and generates an adaptive resource configuration index system. The system maps the original cost influence factors to three types of decision guidance indexes through a composite function:

[0091] ① The storage hierarchy sensitivity index (Si) comprehensively considers the response degree of the data block to the price change of different storage hierarchies, forming a scoring system of 0-100. The data block with a score exceeding 75 is highly sensitive to storage cost and is suitable for priority consideration for migration to a low-cost storage hierarchy;

[0092] ② The access frequency threshold (Ai) is based on data access pattern analysis to determine the optimal storage hierarchy conversion point for each type of data. For example, if the access frequency of a certain type of log data is less than 3 times per week, the system will set its Ai index to "warm storage suitable";

[0093] ③ The migration timing threshold (Ti) integrates the billing period factor and the cost prediction model to provide a basis for judging the best execution window for data migration, such as "7 days before the next billing period" or "after the access frequency drops to the threshold for 15 days".

[0094] The index evaluation module of the present application enables enterprises to develop precise hierarchical storage strategies based on data characteristics and business needs through this scientific evaluation system, ensuring data access performance while significantly reducing storage costs and improving cloud resource utilization efficiency. The evaluation results are presented in the form of a visual dashboard, supporting decision-makers to make intuitive and efficient migration decisions.

[0095] In one embodiment, the decision optimization module includes:

[0096] A function establishment module for establishing a migration target function based on the elastic billing evaluation index, with the goal of minimizing migration cost and combining access performance constraints;

[0097] A timing identification module for calculating the cost-benefit score of performing migration operations at different time points using the migration target function, and identifying the migration time window with the lowest cost;

[0098] The scheduling generation module is configured to determine target migration time and priority ranking of each data block according to the cost-benefit score and migration urgency, and establish a migration schedule table.

[0099] Specifically, the decision optimization module realizes accurate decision of the data migration time by systematically analyzing the flexible billing evaluation index. The function establishment module first constructs a migration target function F(d, t, l), wherein d represents a data block identifier, t represents a migration execution time point, and l represents a target storage level. The core expression of the migration target function is:

[0100] F(d, t, l) = TC(d, t, l) + PC(d, t, l);

[0101] In the formula, TC represents a total cost function, and PC represents a performance compensation function. The total cost function is further expanded as:

[0102] TC(d, t, l) = SC(d, l) + AC(d, l) + MC(d, t) - SR(t);

[0103] In the formula, SC is a storage cost function, which is calculated according to the unit storage price of the target storage level l and the size of the data block d; AC is an access cost function, which is calculated based on the estimated access frequency of the data block d at the target level l and the unit access price; MC is a migration operation cost function, which is related to the data block size and the migration time t; and SR is a step billing saving function, which reflects the billing discount obtained by executing migration at the migration execution time point t.

[0104] The performance compensation function PC is used to quantify the performance constraint condition, and the expression is:

[0105] PC(d, t, l) = λ × max(0, P min (d, l)) - P 2 (d, lcurrent);

[0106] In the formula, P(d, l) represents the access performance index of the data block d at the storage level l, P min is the minimum performance threshold required by the business, and λ is a weight coefficient. When the performance is lower than the threshold, the function value rapidly increases, ensuring that the system preferentially selects a migration scheme that meets the performance requirement.

[0107] Specifically, the time point identification module calculates the cost-benefit score for each time point in the billing period based on the migration target function. The calculation formula of the cost-benefit score CES is:

[0108] CES(d, t, l) = [F(d, 0, lcurrent) - F(d, t, l)] / MC(d, t);

[0109] In the formula, lcurrent represents the current storage level of the data block d, F(d, 0, lcurrent) represents the baseline cost without performing migration, the CES value represents the overall benefit brought by the unit migration cost, and the higher the value, the better the migration opportunity.

[0110] In actual operation, the system samples and evaluates the time points in the future 30 days in each calculation period (such as 24 hours) to generate a CES timing curve. The system automatically identifies a set of local maximum points {t1, t2,..., t n} on the CES curve, which represent potential optimal migration time windows, and eliminates suboptimal time points below a threshold τ to finally determine the optimal migration time window.

[0111] Specifically, the scheduling generation module combines the cost-benefit score with a migration urgency index to determine the migration priority of the data block. The expression of the migration urgency index UM is:

[0112] UM(d) = (1-τ rem / τ total ) x IF(d) x CR(d);

[0113] In the formula, τ rem represents the remaining time from the optimal migration time window, τ total represents the total length of the billing period, IF(d) represents the service importance factor of the data block d (between 0 and 1), and CR(d) represents the storage cost change rate. The higher the UM value, the higher the migration urgency.

[0114] The system sorts all data blocks to be migrated based on the comprehensive score CS = w1 x CES + w2 x UM (where w1 and w2 are weight coefficients and w1 + w2 = 1) to determine the migration priority. The sorting result, together with the optimal migration time window, constitutes a detailed migration opportunity scheduling table, which includes the following fields: data block identifier, source storage level, target storage level, planned migration time, migration priority, estimated cost savings, and performance impact rating. The scheduling table is updated every 24 hours, and will trigger immediate recalculation according to external factors such as changes in cloud resource prices, changes in access patterns, or storage policy adjustments, to ensure that the migration decision always reflects the latest cost optimization opportunities.

[0115] In one embodiment, the batch planning unit 2 includes:

[0116] A demand analysis module for analyzing the time distribution characteristics of the target migration time of each data block through the migration opportunity scheduling table, and constructing a migration demand distribution matrix in combination with the elastic billing evaluation index;

[0117] The constraint evaluation module is configured to analyze a resource constraint influence coefficient according to a real-time bandwidth state and a storage level load condition of the cloud resource platform, and dynamically adjust a migration parameter based on a preset threshold value.

[0118] The batch generation module is configured to generate a migration planning batch based on the migration demand distribution matrix and the resource constraint influence coefficient.

[0119] In one embodiment, the demand analysis module comprises:

[0120] The information extraction module is configured to extract target migration timing and priority ranking information of each data block based on the migration timing schedule.

[0121] The feature recognition module is configured to analyze time distribution characteristics of the target migration timing of each data block, and recognize migration demand density and resource competition degree in a billing period.

[0122] The matrix construction module is configured to construct a migration demand distribution matrix in combination with the migration demand density, the resource competition degree in the billing period, and the storage level sensitivity and access frequency threshold in the elastic billing evaluation index.

[0123] Specifically, the feature recognition module receives the output result of the information extraction module, first divides the entire billing period (such as 30 days) into multiple time slots (such as every 2 hours), and then counts the number of migration tasks, the total data volume, and the average priority in each time slot. Based on these statistical data, the system recognizes the migration demand density variation trend in the billing period, and at the same time, the system calculates the resource competition degree, that is, evaluates the potential contention of multiple migration tasks on network bandwidth, storage IO, and other resources in a specific time period. The output of the feature recognition module is a time series mapping table, which shows the spatiotemporal distribution characteristics of the migration demand in the entire billing period, providing a key basis for the next matrix construction.

[0124] Specifically, the matrix construction module synthesizes the analysis results of the previous two modules, and constructs a multi-dimensional migration demand distribution matrix in combination with the storage level sensitivity and access frequency threshold in the elastic billing evaluation index. The matrix adopts a two-dimensional structure of “time x storage level”, and each cell contains three core attributes: the data volume migrated from a specific source storage level to a specific target storage level in the time period, the priority weighted value, and the resource demand estimation.

[0125] In one embodiment, as Figure 3As shown, when the constraint evaluation module analyzes the resource constraint influence coefficient according to the real-time bandwidth state and storage level load of the cloud resource platform, and dynamically adjusts the migration parameters based on the preset threshold, it includes: calculating the resource constraint influence coefficient; if the network bandwidth availability of the cloud resource platform is lower than the bandwidth safety threshold, then reducing the migration data volume in unit time according to the bandwidth adjustment ratio and correspondingly increasing the migration batch quantity; if the IOPS usage rate of the target storage level exceeds the load upper threshold, then adjusting the migration execution time to the time period when the IOPS usage rate of the storage level is lower than the load suitable threshold; if the access operation quantity of the source storage level increases at a rate exceeding the access fluctuation threshold in the monitoring period, then delaying the non-critical migration task to the next billing period and recalculating the resource constraint influence coefficient.

[0126] Specifically, calculating the resource constraint influence coefficient includes: obtaining the network bandwidth usage, storage level IOPS load level and access operation frequency of the cloud resource platform in real time through the API interface, and solving the resource constraint influence coefficient; the expression of the resource constraint influence coefficient is:

[0127] ;

[0128] In the formula, IC is the resource constraint influence coefficient; BW r is the current network bandwidth availability; BW max is the maximum network bandwidth availability; IOPS c is the current IOPS usage rate of the target storage level; IOPS max is the maximum IOPS threshold of the target storage level; AO r is the growth rate of the access operation quantity of the source storage level; AO max is the maximum allowed value of the access operation growth rate; α 、 β 、 γ are weight coefficients of the network bandwidth factor, the IOPS load factor and the access operation factor respectively, and α + β + γ =1.

[0129] In one embodiment, the batch generation module includes:

[0130] a scheme formulation module for determining the migration priority according to the migration demand distribution matrix, and establishing a batch allocation scheme for each data block in combination with the resource constraint influence coefficient;

[0131] The task merging module is configured to merge the migration tasks in the same billing period in the batch allocation scheme based on the billing period information of the cloud resource platform, so as to reduce the migration operation cost.

[0132] The planning output module is configured to generate a migration planning batch containing a batch sequence, a predicted execution time and a resource demand amount, and to synchronize the migration planning batch with the migration timing schedule in time.

[0133] Specifically, as shown in the figure, Figure 4 The scheme making module first establishes a preliminary batch allocation scheme of the data blocks based on the priority information in the migration demand distribution matrix and in combination with the resource constraint influence coefficient. For example, for the critical business data with a priority score of 90 or above, the system will execute the migration at the original planned migration timing as far as the resource constraint permits; and for the data with a lower priority, the system will appropriately adjust the migration execution time according to the size of the resource constraint influence coefficient, so as to avoid the resource competition peak period. The task merging module further optimizes the preliminary scheme. For example, the system finds that there are 5 data blocks that need to be migrated from the same source storage level to the same target storage level in the same day, and the time windows are close (e.g., the difference is not more than 4 hours), and then the system merges these tasks into one batch, shares the migration resources and reduces the repeated migration operation cost. Finally, the planning output module generates the final migration planning batch, each batch containing a clear batch sequence number, a specific execution time period, a list of data blocks participating in the migration, an estimated resource demand amount (bandwidth, IOPS, CPU, etc.) and a predicted completion time. The migration planning batch is presented in the form of a task Gantt chart and a resource usage prediction curve, which supports the administrator to further manually review and adjust, so as to realize the human-machine collaborative migration management.

[0134] In order to facilitate the understanding of the above technical solutions of the present application, the following will be specifically described as follows by taking the data management system of a regional bank as an example:

[0135] The regional bank has business data scattered in multiple cloud platforms, including transaction records, customer information and operation analysis, etc. With the growth of data volume, the bank faces problems such as rising storage cost and complex multi-cloud management. After deploying the data hierarchical storage migration service system of the present application, the bank realizes intelligent data management based on the elastic billing characteristics of the cloud resources.

[0136] In the cost decision unit 1, the system obtains the price data of different storage levels by calling the cloud platform API interface through the cost modeling module. The system finds that the cloud platform mainly used by the bank has a 8% reduction in monthly storage operation cost at the end of each month, and usually has price fluctuations at the beginning of each quarter. The index evaluation module analyzes the access mode of the bank's transaction data and determines that the time-sensitive data (transaction records in the past 3 months) should be retained in the standard storage layer, while the historical data with decreased access frequency should be migrated to the low-frequency or archival storage layer. Based on these analyses, the decision optimization module generates a migration timing schedule that clearly specifies the best migration time and target storage level for different data.

[0137] In the batch planning unit 2, the demand analysis module finds through analysis of the migration timing schedule that multiple database backups need to be migrated to the archival storage at the beginning of the quarter. The constraint evaluation module monitors that the network bandwidth usage rate reaches 76% during this period, close to the 80% safety threshold set by the system. Accordingly, the resource constraint influence coefficient is calculated to be 0.75, indicating that there is a significant resource limitation. The batch generation module then adjusts the migration strategy by dispersing the migration tasks of large backup files to several consecutive business off-peak periods and combining multiple small migration tasks to improve efficiency.

[0138] In the migration execution unit 3, the system automatically executes data migration operations according to the planned batches within the specified time window. The system migrates the transaction records of the past 12 months from the standard storage to the low-frequency storage, and migrates the historical data two years ago to the archival storage. The entire process is completed automatically without human intervention, and the system monitors the migration progress and resource usage in real time, and dynamically adjusts the execution strategy if necessary.

[0139] After using the system for 6 months, the bank's storage cost has been reduced by 18.5%, the data management personnel's workload has been reduced by 35%, and the system response time has been improved by 12%, fully verifying the practical value and economic benefits of the present invention in the field of data tiered storage migration in the cloud environment.

[0140] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A cloud resource based data tiered storage migration service system, characterized in that, The application comprises: a cost decision unit for establishing a cost-optimized migration decision mechanism for data blocks in different storage levels according to the cloud resource platform billing rules and price data, and determining a migration timing schedule; a batch planning unit for constructing a migration demand distribution matrix and analyzing resource constraint influence coefficients based on the migration timing schedule and the real-time state of the cloud resource platform, and generating a migration planning batch; a migration execution unit for executing migration operations of data blocks in different storage levels according to the migration timing schedule and the migration planning batch; The cost decision unit comprises: a cost modeling module for establishing a multi-dimensional cost calculation model by determining the layered storage cost, data access cost and migration operation cost based on the cloud resource platform billing rules and price data; an index evaluation module for generating an elastic billing evaluation index by analyzing the cost influence factors of data migration using the multi-dimensional cost calculation model; and a decision optimization module for constructing a cost-optimized migration decision function according to the elastic billing evaluation index, and determining the migration timing schedule; The batch planning unit comprises: a demand analysis module for analyzing the time distribution characteristics of the target migration timing of each data block based on the migration timing schedule, and constructing a migration demand distribution matrix in combination with the elastic billing evaluation index; a constraint evaluation module for analyzing the resource constraint influence coefficients based on the real-time bandwidth state and the storage level load of the cloud resource platform, and dynamically adjusting the migration parameters based on a preset threshold; and a batch generation module for generating a migration planning batch based on the migration demand distribution matrix and the resource constraint influence coefficients; The demand analysis module comprises: an information extraction module for extracting the target migration timing and priority sorting information of each data block based on the migration timing schedule; a feature recognition module for analyzing the time distribution characteristics of the target migration timing of each data block, and identifying the migration demand density and resource competition degree in the billing period; and a matrix construction module for constructing the migration demand distribution matrix in combination with the migration demand density, resource competition degree in the billing period, and the storage level sensitivity and access frequency threshold in the elastic billing evaluation index.

2. The cloud resource based data tiering migration service system as claimed in claim 1, wherein, The cost decision unit is connected with the batch planning unit and the migration execution unit; The cost modeling module comprises: a data collection module for obtaining layered storage billing data, data access billing data and migration operation billing data of the cloud resource platform through an API interface, and establishing a multi-dimensional billing data set; a cost analysis module for analyzing the layered storage cost, data access cost and migration operation cost based on the multi-dimensional billing data set, in combination with the different storage level price coefficients and billing period information of the cloud resource platform; a model construction module for establishing a multi-dimensional cost calculation model according to the layered storage cost, data access cost and migration operation cost.

3. The cloud resource based data tiering migration service system as claimed in claim 2, wherein, The analysis of the layered storage cost, data access cost and migration operation cost comprises: based on the layered storage billing data, extracting the unit capacity price, minimum storage duration requirement and early deletion penalty fee of different storage levels, and obtaining the layered storage cost in combination with the actual occupied space and expected retention period of the data block; According to the read-write operation unit price, data retrieval cost and bandwidth rate in the data access billing data, combined with historical access mode and access frequency statistical information, the data access cost is calculated; The data transmission cost, computing resource occupation cost and cross-region additional cost in the migration operation billing data are utilized to analyze the billing low period and peak period in different migration periods, and the migration operation cost is determined.

4. The cloud resource based data tiering migration service system as claimed in claim 2, wherein, The index evaluation module comprises: The cost estimation module is configured to determine the current storage cost, access cost and operation cost of potential migration paths of each data block distributed in different storage levels based on a multi-dimensional cost calculation model and historical storage records of the cloud resource platform; The factor extraction module is configured to calculate the cost change rate, resource utilization efficiency and billing period coefficient under different migration strategies based on the current storage cost, access cost and operation cost of potential migration paths of each data block, and generate key cost influence factors; The index construction module is configured to establish an elastic billing evaluation index including storage level sensitivity, access frequency threshold and migration timing critical value based on the key cost influence factors.

5. The cloud resource based data tiering migration service system as claimed in claim 2, wherein, The decision optimization module comprises: The function establishment module is configured to establish a migration target function based on the elastic billing evaluation index and taking the minimization of migration cost as the target, and combined with access performance constraint conditions; The timing identification module is configured to calculate the cost-benefit score of performing migration operation at different time points by using the migration target function, and identify the migration time window with the lowest cost; The scheduling generation module is configured to determine the target migration timing and priority ranking of each data block according to the cost-benefit score and migration urgency, and establish a migration scheduling table.

6. The cloud resource based data tiering migration service system as claimed in claim 1, wherein, The migration parameters are dynamically adjusted based on the preset threshold according to the real-time bandwidth state and storage level load of the cloud resource platform, and the resource constraint influence coefficient is analyzed, which comprises: Calculating the resource constraint influence coefficient; If the network bandwidth availability of the cloud resource platform is lower than the bandwidth safety threshold, the migration data volume in unit time is reduced by the bandwidth adjustment ratio and the migration batch quantity is increased accordingly; If the IOPS usage rate of the target storage level exceeds the load upper threshold, the migration execution time is adjusted to a time period in which the IOPS usage rate of the storage level is lower than the load suitable threshold; If the access operation quantity of the source storage level increases by more than the access fluctuation threshold in the monitoring period, the non-critical migration task is delayed to the next billing period and the resource constraint influence coefficient is recalculated.

7. The cloud resource based data tiering migration service system as claimed in claim 6, wherein, The calculation of the resource constraint influence coefficient comprises: The network bandwidth usage, storage level IOPS load level and access operation frequency of the cloud resource platform are obtained in real time through an API interface, and the resource constraint influence coefficient is solved; The expression of the resource constraint influence coefficient is: ； In the formula, IC is a resource constraint impact coefficient; BW r is a current network bandwidth availability rate; BW max is a maximum network bandwidth availability rate; IOPS c is a target storage tier current IOPS usage rate; IOPS max is a target storage tier maximum IOPS threshold; Δ AO r is a source storage tier access operation quantity growth rate; AO max is a maximum allowed access operation growth rate; α , β , γ are weight coefficients of the network bandwidth factor, the IOPS load factor, and the access operation factor, respectively.

8. The cloud resource based data tiering migration service system as claimed in claim 1, wherein, The batch generation module comprises: The scheme development module is configured to determine the migration priority according to the migration demand distribution matrix, and establish a batch allocation scheme of each data block combined with the resource constraint influence coefficient; The task merging module is configured to merge the migration tasks in the same billing period in the batch allocation scheme based on the billing period information of the cloud resource platform, so as to reduce the migration operation cost. The planning output module is configured to generate a migration planning batch containing a batch sequence, an estimated execution time, and a resource requirement amount, and to synchronize the migration planning batch with a migration timing schedule in time.

Images