Data hierarchical storage migration service system based on cloud resources

Abstract

The invention discloses a data hierarchical storage migration service system based on cloud resources, and relates to the technical field of data migration, and the system comprises a cost decision-making unit which is used for establishing a cost optimization migration decision-making mechanism for data blocks in different storage hierarchies according to charging rules and price data of a cloud resource platform, and determining a migration opportunity scheduling table; the batch planning unit is used for constructing a migration demand distribution matrix and analyzing a resource constraint influence coefficient based on the migration opportunity scheduling table and the real-time state of the cloud resource platform, and generating a migration planning batch; and the migration execution unit is used for executing migration operation of the data blocks in the different storage hierarchies according to the migration opportunity scheduling table and the migration planning batch. According to the method, the migration decision can be dynamically optimized according to the elastic charging characteristics of the cloud resources, and the problems that in the prior art, deep utilization of the elastic charging characteristics of the cloud resources is lacked, and intelligent planning of migration opportunities and batches cannot be achieved are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data migration, in particular to a data hierarchical storage migration service system based on cloud resources. BACKGROUND

[0002] With the rapid development of cloud computing technology, enterprise data storage needs are showing a trend of diversification, and data hierarchical storage technology has emerged as the times require. The data hierarchical storage migration service system based on cloud resources has become a key requirement for enterprise data management, which can dynamically allocate data of different temperatures to storage levels that match performance and cost, and achieve optimal configuration of storage resources. However, when enterprise data is scattered in multi-cloud and hybrid cloud environments, how to perform cost-optimized data migration according to the elastic billing characteristics of cloud resources has become a technical problem that needs to be solved urgently, and enterprises need a system that can identify the best migration opportunity and optimize the migration batch to achieve cross-platform, efficient and low-cost data management.

[0003] At present, there are many data storage and migration solutions on the market, mainly including data migration services provided by cloud service providers and third-party data management platforms. These tools provide basic data transfer functions and simple hierarchical storage strategies, such as automatically adjusting data storage levels according to access frequency; at the same time, some hybrid cloud data management platforms support cross-cloud data operations, providing basic support for enterprise data governance. These technical solutions have alleviated the pressure of enterprise data management to a certain extent, provided technical support for enterprise data governance, and promoted the process of migrating enterprise IT infrastructure to the cloud.

[0004] Although the existing technology has made certain progress in the field of data migration and hierarchical storage, there are still many deficiencies. First of all, most solutions only focus on data management within a single cloud platform, cannot adapt to the elastic billing mode of cloud storage, and cannot dynamically adjust the migration decision according to the real-time pricing strategy; secondly, the existing system lacks intelligent judgment of the migration opportunity and cannot identify the best migration window within the billing period, resulting in increased migration costs; thirdly, traditional tools do not consider resource constraints in the cloud environment and cannot dynamically adjust the migration batch according to network bandwidth and storage performance, and lack comprehensive consideration of multi-dimensional costs of storage, access and migration.

[0005] In view of the problems in the related art, no effective solution has been proposed so far. SUMMARY

[0006] In view of the problems in the prior art, the application provides a data hierarchical storage migration service system based on cloud resources, which has the advantages of dynamically optimizing migration decisions according to the characteristics of cloud resources, and further solves the problems that the elastic billing characteristics of cloud resources are not fully utilized in the prior art, migration decisions cannot be optimized according to price changes, and intelligent planning of migration timing and batches cannot be achieved.

[0007] To this end, the application adopts the following specific technical solutions: The data hierarchical storage migration service system based on cloud resources comprises: A cost decision unit configured to establish a cost-optimized migration decision mechanism for data blocks in different storage levels according to billing rules and price data of a cloud resource platform, and determine a migration timing schedule; A batch planning unit configured to construct a migration demand distribution matrix and analyze resource constraint influence coefficients based on the migration timing schedule and real-time state of the cloud resource platform, and generate a migration planning batch; A migration execution unit configured to execute migration operations of data blocks in different storage levels according to the migration timing schedule and the migration planning batch; The cost decision unit comprises: A cost modeling module configured to establish a multi-dimensional cost calculation model by determining hierarchical storage costs, data access costs and migration operation costs based on billing rules and price data of the cloud resource platform; An index evaluation module configured to analyze cost influence factors of data migration by using the multi-dimensional cost calculation model, and generate elastic billing evaluation indexes; A decision optimization module configured to construct a cost-optimized migration decision function according to the elastic billing evaluation indexes, and determine the migration timing schedule.

[0008] Further, the cost decision unit is connected with the batch planning unit and the migration execution unit; and the cost modeling module comprises: A data acquisition module configured to acquire hierarchical storage billing data, data access billing data and migration operation billing data of the cloud resource platform through an API interface, and establish a multi-dimensional billing data set; A cost analysis module configured to analyze hierarchical storage costs, data access costs and migration operation costs based on the multi-dimensional billing data set, in combination with different storage level price coefficients and billing period information of the cloud resource platform; A model construction module configured to establish a multi-dimensional cost calculation model according to the hierarchical storage costs, the data access costs and the migration operation costs.

[0009] Further, the analysis of the hierarchical storage cost, data access cost and migration operation cost comprises: based on the hierarchical storage billing data, extracting the unit capacity price, minimum storage time length requirement and early deletion penalty fee of different storage levels, combining the actual occupied space of the data block and the expected retention period, obtaining the hierarchical storage cost; according to the read-write operation unit price, data retrieval fee and bandwidth rate in the data access billing data, combining the historical access mode and access frequency statistical information, the data access cost is calculated; using the data transmission fee, computing resource occupation fee and cross-region additional fee in the migration operation billing data, analyzing the billing low period and peak period in different migration periods, and determining the migration operation cost.

[0010] Further, the index evaluation module comprises: a cost estimation module, configured to determine the current storage cost, access cost and operation cost of potential migration paths of each data block distributed in different storage levels based on a multi-dimensional cost calculation model and in combination with historical storage records of the cloud resource platform; a factor extraction module, configured to calculate the cost change rate, resource utilization efficiency and billing period coefficient under different migration strategies according to the current storage cost, access cost and operation cost of potential migration paths of each data block, and generate key cost influence factors; an index construction module, configured to establish an elastic billing evaluation index including storage level sensitivity, access frequency threshold and migration timing critical value based on the key cost influence factors.

[0011] Further, the decision optimization module comprises: a function establishment module, configured to establish a migration target function based on the elastic billing evaluation index and with the minimization of migration cost as the target and in combination with access performance constraint conditions; a timing identification module, configured to calculate the cost-benefit score of performing migration operation at different time points by using the migration target function, and identify the migration time window with the lowest cost; a scheduling generation module, configured to determine the target migration timing and priority ranking of each data block according to the cost-benefit score and migration urgency, and establish a migration scheduling table.

[0012] Further, the batch planning unit comprises: a demand analysis module, configured to analyze the time distribution characteristics of the target migration timing of each data block through the migration timing scheduling table, and construct a migration demand distribution matrix in combination with the elastic billing evaluation index; a constraint evaluation module, configured to analyze the resource constraint influence coefficient according to the real-time bandwidth state and storage level load of the cloud resource platform, and dynamically adjust the migration parameters based on a preset threshold; a batch generation module, configured to generate a migration planning batch based on the migration demand distribution matrix and the resource constraint influence coefficient.

[0013] Further, the demand analysis module comprises: an information extraction module configured to extract target migration time and priority ranking information of each data block based on the migration time scheduling table; a feature recognition module configured to analyze time distribution characteristics of the target migration time of each data block, and recognize migration demand density and resource competition degree in the billing period; a matrix construction module configured to construct a migration demand distribution matrix in combination with the migration demand density, the resource competition degree in the billing period, and the storage level sensitivity and access frequency threshold in the elastic billing evaluation index.

[0014] Further, according to the real-time bandwidth state and storage level load of the cloud resource platform, the resource constraint influence coefficient is analyzed, and the migration parameters are dynamically adjusted based on the preset threshold, including: calculating the resource constraint influence coefficient; if the network bandwidth availability of the cloud resource platform is lower than the bandwidth safety threshold, then the migration data volume in unit time is reduced according to the bandwidth adjustment ratio and the migration batch quantity is increased accordingly; if the IOPS usage rate of the target storage level exceeds the load upper threshold, then the migration execution time is adjusted to a time period in which the IOPS usage rate of the storage level is lower than the load suitable threshold; if the access operation quantity of the source storage level increases at a rate exceeding the access fluctuation threshold in the monitoring period, then the non-critical migration task is delayed to the next billing period and the resource constraint influence coefficient is recalculated.

[0015] Further, the calculation of the resource constraint influence coefficient comprises: acquiring the network bandwidth usage, the storage level IOPS load level and the access operation frequency of the cloud resource platform in real time through an API interface, and solving the resource constraint influence coefficient; the expression of the resource constraint influence coefficient is: ; In the formula, IC is the resource constraint influence coefficient; BW r is the current network bandwidth availability; BW max is the maximum network bandwidth availability; IOPS c is the current IOPS usage rate of the target storage level; IOPS max is the maximum IOPS threshold of the target storage level; Δ AO r is the growth rate of the access operation quantity of the source storage level; AO max is the maximum allowed value of the access operation growth rate; α 、 β 、 γThe weight coefficients of the network bandwidth factor, the IOPS load factor and the access operation factor respectively.

[0016] Further, the batch generation module comprises: The scheme formulation module is configured to determine a migration priority according to the migration demand distribution matrix, and establish a batch allocation scheme of each data block in combination with the resource constraint influence coefficient. The task merging module is configured to merge the migration tasks in the same billing period in the batch allocation scheme based on the billing period information of the cloud resource platform, so as to reduce the migration operation cost. The planning output module is configured to generate a migration planning batch comprising a batch sequence, a predicted execution time and a resource demand amount, and synchronize the migration planning batch with the migration timing schedule in time.

[0017] The present application has the following advantages: (1) The present application establishes a multi-dimensional cost calculation model based on the elastic billing characteristics of the cloud resource platform, and combines the storage level sensitivity, the access frequency threshold and the migration timing critical value and other elastic billing evaluation indexes, so that the system can accurately identify the cost-optimal migration window in the billing period, and dynamically adjust the migration batch according to the resource constraint influence coefficient, thereby effectively reducing the total data storage cost of the enterprise in the multi-cloud environment. According to tests, the cost can be saved by 15%-30% in typical application scenarios, and the cost optimization effect will be more significant as the data size increases and the cloud resource price fluctuates.

[0018] (2) The present application is based on the dual constraint mechanism of the migration demand distribution matrix and the resource constraint influence coefficient, and the system can intelligently identify the migration demand density and the resource competition degree in the billing period, optimize and integrate the migration tasks in the same billing period through the task merging module, and automatically adjust the migration parameters when the key resource indexes such as network bandwidth, storage IOPS and access operation approach the threshold value. Not only does it ensure business continuity and system stability, but also maximizes resource utilization through intelligent planning and dynamic adjustment of migration batches, significantly improves data migration efficiency and reduces management complexity. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a principle block diagram of a data hierarchical storage migration service system based on cloud resources according to an embodiment of the present application; Figure 2is a specific implementation diagram of a cost decision unit in a cloud resource-based data tiered storage migration service system according to an embodiment of the present application; Figure 3 is a specific implementation diagram of a constraint evaluation module in a cloud resource-based data tiered storage migration service system according to an embodiment of the present application; Figure 4 is a specific implementation diagram of a batch generation module in a cloud resource-based data tiered storage migration service system according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] To further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can be used to explain the operating principle of the embodiments in conjunction with the related description of the specification. Those of ordinary skill in the art should be able to understand other possible implementations and advantages of the present application in conjunction with reference to these contents. The components in the drawings are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0022] According to an embodiment of the present application, a cloud resource-based data tiered storage migration service system is provided.

[0023] The present application will be further described in conjunction with the drawings and specific embodiments. As shown in Figure 1 According to an embodiment of the present application, a cloud resource-based data tiered storage migration service system is provided, which includes: a cost decision unit 1 for establishing a cost optimization migration decision mechanism for data blocks in different storage tiers according to cloud resource platform billing rules and price data, and determining a migration timing schedule; a batch planning unit 2 for constructing a migration demand distribution matrix and analyzing resource constraint influence coefficients based on the migration timing schedule and real-time state of the cloud resource platform, and generating a migration planning batch; a migration execution unit 3 for executing migration operations of data blocks in different storage tiers according to the migration timing schedule and the migration planning batch; Specifically, the cost decision unit 1 is connected to the batch planning unit 2 and the migration execution unit 3.

[0024] In one embodiment, as shown in Figure 2 the cost decision unit 1 includes: a cost modeling module for establishing a multi-dimensional cost calculation model by determining tiered storage costs, data access costs, and migration operation costs based on cloud resource platform billing rules and price data; an index evaluation module for analyzing cost influence factors of data migration using the multi-dimensional cost calculation model to generate elastic billing evaluation indexes; A decision optimization module is configured to construct a cost-optimized migration decision function according to the elastic billing evaluation index, and determine a migration timing schedule.

[0025] In one embodiment, the cost modeling module comprises: A data acquisition module is configured to acquire hierarchical storage billing data, data access billing data and migration operation billing data of a cloud resource platform through an API interface, and establish a multi-dimensional billing data set; A cost analysis module is configured to analyze hierarchical storage cost, data access cost and migration operation cost based on the multi-dimensional billing data set, in combination with different storage level price coefficients and billing period information of the cloud resource platform; A model construction module is configured to establish a multi-dimensional cost calculation model according to the hierarchical storage cost, data access cost and migration operation cost.

[0026] In one embodiment, analyzing the hierarchical storage cost, data access cost and migration operation cost comprises: based on the hierarchical storage billing data, extracting the unit capacity price, minimum storage duration requirement and early deletion penalty fee of different storage levels, obtaining the hierarchical storage cost in combination with the actual occupied space of the data block and the expected retention period; according to the read-write operation unit price, data retrieval fee and bandwidth rate in the data access billing data, combining the historical access mode and access frequency statistical information, the data access cost is calculated; using the data transmission fee, computing resource occupation fee and cross-region additional fee in the migration operation billing data, analyzing the billing low valley period and peak period of different migration periods, and determining the migration operation cost.

[0027] Specifically, the present application realizes accurate calculation and prediction of multi-level storage migration cost in a cloud environment through the cost modeling module. In the actual implementation process, the module first connects the open API interfaces of major cloud service providers (such as Ali Cloud, Tencent Cloud, AWS, etc.) through the data acquisition module, and regularly (for example, every 6 hours) acquires the latest billing rules and price parameters. The system will convert these raw data into standardized JSON format, and construct a multi-dimensional billing data set containing three dimensions (storage, access, migration), which is convenient for subsequent analysis and processing.

[0028] Specifically, after obtaining the multi-dimensional billing data set, the cost analysis module will conduct in-depth cost analysis. Taking the tiered storage cost as an example, the system will extract the unit capacity price (such as 0.12 yuan per GB per month), the minimum storage period requirement (such as 30 days, 90 days, 180 days, etc.), and the pre-removal penalty fee for different storage tiers (such as hot storage, warm storage, cold storage, archive storage, etc.). The system will then calculate the storage cost of each data block in different storage tiers based on the actual occupied space size and the estimated retention period of each data block in the actual business scenario of the enterprise. For example, for a 100 GB data block with an estimated retention period of 45 days, the system will calculate the cost difference between the hot storage tier (no minimum period limit) and the warm storage tier (assuming a minimum of 30 days), including possible pre-removal penalty fees. Secondly, for data access cost, the system calculates the access cost of each storage tier based on the read and write operation price, request fee and data transfer rate in the cloud platform billing data, combined with the historical access records of the data block. Finally, the migration operation cost analysis mainly focuses on data transfer fees, computing resource usage fees and possible cross-region additional fees. The system will identify the low and peak periods in the cloud platform billing period to provide a basis for subsequent migration batch planning. For example, the data transfer fee of a certain cloud service provider may be 30% lower during the hours of 2am to 6am than during working hours. The system will incorporate this information into the multi-dimensional cost calculation model to optimize the migration timing selection.

[0029] Specifically, the model construction module establishes a multi-dimensional cost calculation model based on the above three types of cost analysis results. The model represents the cost relationship of different storage tiers, different data blocks, and different time points in matrix form, and the expression is: C(b, l, t) = λ × S(b, l) + μ × A(b, l, t) + ω × M(b, l1, l2, t); In the formula, C represents the total cost function, b represents the data block, l represents the storage tier, t represents the migration execution time point, S represents the storage cost function, A represents the access cost function, M represents the migration cost function from the l1 tier to the l2 tier, and λ, μ, ω are the weight coefficients of the three types of costs, which can be dynamically adjusted according to the business emphasis of different data.

[0030] The multi-dimensional cost calculation model not only considers the static storage cost, but also includes the dynamic access cost and migration cost, so that the system can comprehensively evaluate the economic efficiency of different migration strategies in the subsequent decision-making process, and provide a cost-optimal solution for data migration.

[0031] In one embodiment, the index evaluation module includes: a cost estimation module, configured to determine, based on a multi-dimensional cost calculation model and in combination with historical storage records of the cloud resource platform, current storage costs, access costs and operation costs of potential migration paths of each data block distributed in different storage levels; a factor extraction module, configured to calculate, according to the current storage costs, the access costs and the operation costs of potential migration paths of each data block, cost change rates, resource utilization efficiencies and billing period coefficients under different migration strategies, and generate key cost influence factors; an index construction module, configured to establish, based on the key cost influence factors, elastic billing evaluation indexes including storage level sensitivity, access frequency threshold and migration timing critical value.

[0032] Specifically, the index evaluation module of the present application realizes accurate evaluation and decision optimization of data migration in a cloud environment through a systematic cost analysis method. In actual implementation, the module first comprehensively evaluates enterprise data by using the cost estimation module. Based on the aforementioned multi-dimensional cost calculation model, the module generates detailed cost portraits for each data block in combination with historical storage and access records of the cloud platform in the last 90 days.

[0033] Specifically, the cost estimation module obtains actual cost data of each data block in different storage levels by calling a storage analysis API of the cloud resource platform. For example, for a 500GB enterprise financial data set, the system calculates its monthly storage cost in the current hot storage level (e.g. 500GB x 0.15 yuan / GB = 75 yuan), access cost (e.g. 10,000 requests per month on average x 0.01 yuan / time = 100 yuan), and potential operation cost of migrating it to the cold storage level (e.g. 500GB x 0.08 yuan / GB migration fee = 40 yuan). The system will simultaneously evaluate multiple possible migration paths (e.g. hot storage → warm storage → cold storage or hot storage → archive storage, etc.) and calculate the complete cost structure of each path to form a multi-dimensional cost matrix.

[0034] Specifically, after obtaining detailed cost data, the factor extraction module first calculates the cost change rate under different migration strategies, i.e. the percentage change of total cost before and after migration, expressed as: ΔC%=(C_after-C_before) / C_before×100%; In the formula, C_after represents the total cost after data migration, C_before represents the total cost before data migration, ΔC% represents the cost change percentage caused by the migration strategy, and a positive value represents cost increase and a negative value represents cost saving.

[0035] Then, the resource utilization efficiency is evaluated, and the resource utility coefficient is calculated by analyzing the matching degree of the storage space utilization rate and the access frequency of the data block: η = (S_utilization x A_frequency) / S_capacity; where S_utilization represents the actual storage utilization of the data block, A_frequency represents the matching degree of data access frequency and storage hierarchy performance (value range 0-1), S_capacity represents the total storage capacity allocated to the data block, η represents the resource utility coefficient, and the higher the value, the higher the resource utilization efficiency.

[0036] Finally, the billing cycle coefficient is analyzed, which reflects the matching degree of different storage hierarchy billing cycles and data life cycle: τ = min(T_billing / T_lifecycle, 1) x f(T_remainder); where T_billing represents the billing cycle of the storage hierarchy (e.g., month, quarter, year), T_lifecycle represents the expected life cycle of the data, T_remainder represents the remaining billing cycle time, f() is a smoothing function, and τ represents the billing cycle coefficient, and the closer the value is to 1, the higher the matching degree of the billing cycle and the data life cycle, which helps to reduce storage costs. By analyzing these three dimensions comprehensively, the system extracts a set of key cost impact factors for data migration decisions, providing a basis for subsequent evaluation index construction.

[0037] Specifically, the index construction module constructs an elastic billing multi-dimensional evaluation model based on the extracted key cost impact factors, generating an adaptive resource allocation index system. This system maps the original cost impact factors to three types of decision guidance indicators through a composite function: ① The storage hierarchy sensitivity index (Si) considers the response of data blocks to price changes in different storage hierarchies, forming a scoring system of 0-100. Data blocks with scores exceeding 75 are highly sensitive to storage costs and are suitable for priority consideration for migration to low-cost storage hierarchies; ② The access frequency threshold (Ai) is based on data access pattern analysis to determine the optimal storage hierarchy transition point for each type of data. For example, if the access frequency of a certain type of log data is less than 3 times per week, the system will set its Ai indicator to "warm storage suitable"; ③ The migration timing threshold (Ti) integrates billing cycle factors and cost prediction models to provide judgment basis for the best execution window of data migration, such as "7 days before the next billing cycle" or "access frequency drops below threshold for 15 consecutive days".

[0038] The index evaluation module of the present application can make enterprises formulate accurate hierarchical storage strategies based on data characteristics and business requirements through the scientific evaluation system, significantly reduce storage costs while ensuring data access performance, and improve cloud resource utilization efficiency. The evaluation results are presented in the form of a visual dashboard to support decision-makers to make intuitive and efficient migration decisions.

[0039] In one embodiment, the decision optimization module comprises: a function establishment module for establishing a migration target function based on the elastic billing evaluation index, with the goal of minimizing migration cost and combining access performance constraints; a timing identification module for calculating the cost-benefit score of performing migration operations at different time points using the migration target function, and identifying the migration time window with the lowest cost; a scheduling generation module for determining the target migration timing and priority ranking of each data block according to the cost-benefit score and migration urgency, and establishing a migration schedule table.

[0040] Specifically, the decision optimization module realizes accurate decision-making of data migration timing through systematic analysis of the elastic billing evaluation index. The function establishment module first constructs the migration target function F(d, t, l), where d represents the data block identifier, t represents the migration execution time point, and l represents the target storage level. The core expression of the migration target function is: F(d, t, l) = TC(d, t, l) + PC(d, t, l); In the formula, TC represents the total cost function, and PC represents the performance compensation function. The total cost function is further expanded as: TC(d, t, l) = SC(d, l) + AC(d, l) + MC(d, t) - SR(t); In the formula, SC is the storage cost function, which is calculated according to the unit storage price of the target storage level l and the size of the data block d; AC is the access cost function, which is calculated based on the estimated access frequency of the data block d at the target level l and the unit access price; MC is the migration operation cost function, which is related to the data block size and migration time t; SR is the step billing saving function, which reflects the billing discount obtained by performing migration at the migration execution time point t.

[0041] The performance compensation function PC is used to quantify the performance constraints, and the expression is: PC(d, t, l) = λ × max(0, P min -P(d, l)) 2 ; In the formula, P(d, l) represents the access performance index of the data block d at the storage level l, P minis the minimum performance threshold for business requirements, and λ is the weight coefficient. When the performance is lower than the threshold, the function value increases rapidly, ensuring that the system prioritizes the migration scheme that meets the performance requirements.

[0042] Specifically, the opportunity identification module calculates the cost-benefit score for each time point in the billing period based on the migration target function. The calculation formula of the cost-benefit score CES is: CES(d, t, l) = [F(d, 0, lcurrent) - F(d, t, l)] / MC(d, t); In the formula, lcurrent represents the current storage level of the data block d, F(d, 0, lcurrent) represents the baseline cost without performing migration, the CES value represents the overall benefit brought by unit migration cost, and the higher the value, the better the migration opportunity.

[0043] In actual operation, the system samples and evaluates the time points in the future 30 days within each calculation period (such as 24 hours) to generate a CES time sequence curve. The system automatically identifies the local maximum point set {t1, t2,..., t n} on the CES curve, which represents the potential best migration time window. The suboptimal time points lower than the threshold τ are eliminated, and the best migration time window is finally determined.

[0044] Specifically, the scheduling generation module combines the cost-benefit score with the migration urgency index to determine the migration priority of the data block. The expression of the migration urgency index UM is: UM(d) = (1 - τ rem / τ total ) × IF(d) × CR(d); In the formula, τ rem represents the remaining time from the best migration time window, τ total represents the total length of the billing period, IF(d) represents the business importance factor of the data block d (between 0 and 1), and CR(d) represents the storage cost change rate. The higher the UM value, the higher the migration urgency.

[0045] The system sorts all the data blocks to be migrated based on the comprehensive score CS = w1 × CES + w2 × UM (where w1 and w2 are weight coefficients and w1 + w2 = 1) to determine the migration priority. The sorting result, together with the best migration time window, forms a detailed migration opportunity scheduling table, which contains the following fields: data block identifier, source storage level, target storage level, planned migration time, migration priority, estimated cost saving, and performance impact rating. The scheduling table is updated every 24 hours, and it will trigger immediate recalculation according to external factors such as changes in cloud resource prices, changes in access patterns, or storage policy adjustments, ensuring that the migration decision always reflects the latest cost optimization opportunities.

[0046] In one embodiment, the batch planning unit 2 comprises: a demand analysis module for analyzing the time distribution characteristics of the target migration time of each data block through the migration time scheduling table, and constructing a migration demand distribution matrix in combination with the elastic billing evaluation index; a constraint evaluation module for analyzing the resource constraint influence coefficient according to the real-time bandwidth state and storage level load of the cloud resource platform, and dynamically adjusting the migration parameters based on the preset threshold; a batch generation module for generating a migration planning batch based on the migration demand distribution matrix and the resource constraint influence coefficient.

[0047] In one embodiment, the demand analysis module comprises: an information extraction module for extracting the target migration time and priority ranking information of each data block based on the migration time scheduling table; a feature recognition module for analyzing the time distribution characteristics of the target migration time of each data block, and identifying the migration demand density and resource competition degree in the billing period; a matrix construction module for constructing a migration demand distribution matrix in combination with the migration demand density, resource competition degree, and storage level sensitivity and access frequency threshold in the elastic billing evaluation index in the billing period.

[0048] Specifically, the feature recognition module receives the output results of the information extraction module, first divides the entire billing period (such as 30 days) into multiple time slots (such as every 2 hours), and then counts the number of migration tasks, total data volume, and average priority in each time slot. Based on these statistical data, the system identifies the migration demand density trend in the billing period, and at the same time, the system calculates the resource competition degree, i.e., evaluates the potential contention of multiple migration tasks for network bandwidth, storage IO, etc. in a specific time period. The output of the feature recognition module is a time series mapping table, which shows the spatiotemporal distribution characteristics of the migration demand in the entire billing period, providing a key basis for the next matrix construction.

[0049] Specifically, the matrix construction module synthesizes the analysis results of the previous two modules, and constructs a multi-dimensional migration demand distribution matrix in combination with the storage level sensitivity and access frequency threshold in the elastic billing evaluation index. The matrix adopts a two-dimensional structure of "time x storage level", and each cell contains three core attributes: the data volume migrated from a specific source storage level to a specific target storage level in this time period, the priority weighted value, and the resource demand estimate.

[0050] In one embodiment, as Figure 3As shown, when the constraint evaluation module analyzes the resource constraint influence coefficient according to the real-time bandwidth state and storage level load of the cloud resource platform, and dynamically adjusts the migration parameters based on the preset threshold, it includes: calculating the resource constraint influence coefficient; if the network bandwidth availability of the cloud resource platform is lower than the bandwidth safety threshold, then reducing the migration data volume in unit time according to the bandwidth adjustment ratio and correspondingly increasing the migration batch quantity; if the IOPS usage rate of the target storage level exceeds the load upper threshold, then adjusting the migration execution time to the time period when the IOPS usage rate of the storage level is lower than the load suitable threshold; if the access operation quantity of the source storage level increases at a rate exceeding the access fluctuation threshold in the monitoring period, then delaying the non-critical migration task to the next billing period and recalculating the resource constraint influence coefficient.

[0051] Specifically, calculating the resource constraint influence coefficient includes: obtaining the network bandwidth usage, storage level IOPS load level and access operation frequency of the cloud resource platform in real time through the API interface, and solving the resource constraint influence coefficient; the expression of the resource constraint influence coefficient is: ; In the formula, IC is the resource constraint influence coefficient; BW r is the current network bandwidth availability; BW max is the maximum network bandwidth availability; IOPS c is the current IOPS usage rate of the target storage level; IOPS max is the maximum IOPS threshold of the target storage level; AO r is the growth rate of the access operation quantity of the source storage level; AO max is the maximum allowed value of the access operation growth rate; α 、 β 、 γ are weight coefficients of the network bandwidth factor, the IOPS load factor and the access operation factor respectively, and α + β + γ =1.

[0052] In one embodiment, the batch generation module includes: a scheme formulation module for determining migration priorities according to the migration demand distribution matrix, and establishing a batch allocation scheme for each data block in combination with the resource constraint influence coefficient; a task merging module for merging migration tasks in the same billing period in the batch allocation scheme based on the billing period information of the cloud resource platform, so as to reduce the migration operation cost; The planning output module is configured to generate a migration planning batch containing a batch sequence, an estimated execution time, and a resource requirement amount, and to synchronize the migration planning batch with the migration timing schedule in time.

[0053] Specifically, as shown in Figure 4 the scheme formulation module first establishes a preliminary batch allocation scheme for the data blocks based on the priority information in the migration requirement distribution matrix and in combination with the resource constraint influence coefficient. For example, for critical business data with a priority score of 90 points or above, the system will perform migration at the originally planned migration timing as far as the resource constraints permit; and for data with a lower priority, the system will adjust the migration execution time according to the size of the resource constraint influence coefficient to avoid resource competition peak periods. The task merging module further optimizes the preliminary scheme. For example, the system finds that five data blocks need to be migrated from the same source storage hierarchy to the same target storage hierarchy within the same day, and the time windows are close (e.g., differ by no more than 4 hours), and then merges these tasks into one batch to share migration resources and reduce repeated migration operation costs. Finally, the planning output module generates the final migration planning batch, each batch containing a clear batch sequence number, a specific execution time period, a list of data blocks participating in migration, an estimated resource requirement amount (bandwidth, IOPS, CPU, etc.), and a predicted completion time. The migration planning batch is presented in the form of a task Gantt chart and a resource usage prediction curve, supporting further manual review and adjustment by the administrator to realize human-machine collaborative migration management.

[0054] In order to facilitate understanding of the above technical solutions of the present application, the following will be specifically described as follows by taking a data management system of a certain regional bank as an example: The regional bank has business data scattered in multiple cloud platforms, including transaction records, customer information, and operation analysis, etc. With the growth of data volume, the bank faces problems such as rising storage costs and complex multi-cloud management. After deploying the data hierarchical storage migration service system of the present application, the bank realizes intelligent data management based on the elastic billing characteristics of cloud resources.

[0055] In terms of the cost decision unit 1, the system obtains the price data of different storage hierarchies by calling the cloud platform API interface through the cost modeling module. The system finds that the cloud platform mainly used by the bank has a 8% reduction in monthly end storage operation cost, and there is usually a price fluctuation at the beginning of each quarter. The index evaluation module analyzes the access mode of the bank's transaction data and determines that the time-sensitive data (transaction records in the last 3 months) should be kept in the standard storage layer, while the historical data with decreased access frequency should be migrated to the low-frequency or archival storage layer. Based on these analyses, the decision optimization module generates a migration timing schedule, which clearly defines the best migration time and target storage hierarchy for different data.

[0056] In the batch planning unit 2, the demand analysis module finds that multiple database backups need to be migrated to the archive storage at the same time at the beginning of the quarter by analyzing the migration timing schedule. The constraint evaluation module monitors that the network bandwidth usage rate reaches 76% during this period, close to the 80% safety threshold set by the system. Accordingly, the resource constraint influence coefficient is calculated as 0.75, indicating that there is obvious resource limitation. The batch generation module adjusts the migration strategy accordingly, disperses the migration tasks of large backup files to the business low-peak period of several consecutive days, and combines multiple small migration tasks to improve efficiency.

[0057] In the migration execution unit 3, the system automatically executes the data migration operation according to the planned batch at the specified time window. The system migrates the transaction records of the past 12 months from the standard storage to the low-frequency storage, and migrates the historical data of two years ago to the archive storage. The whole process is completed automatically without human intervention, and the system monitors the migration progress and resource usage in real time, and dynamically adjusts the execution strategy when necessary.

[0058] After using the system for 6 months, the bank's storage cost is reduced by 18.5%, the workload of data management personnel is reduced by 35%, and the system response time is improved by 12%, fully verifying the practical value and economic benefit of the present application in the field of data tiered storage migration in the cloud environment.

[0059] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A cloud resource based data tiered storage migration service system, characterized in that, The application relates to a cloud resource platform data block migration method and device. The application comprises: a cost decision unit for establishing a cost-optimized migration decision mechanism for data blocks in different storage levels according to cloud resource platform billing rules and price data, and determining a migration timing schedule; a batch planning unit for constructing a migration demand distribution matrix and analyzing resource constraint influence coefficients based on the migration timing schedule and real-time state of the cloud resource platform, and generating a migration planning batch; a migration execution unit for executing migration operations of data blocks in different storage levels according to the migration timing schedule and the migration planning batch. The cost decision unit comprises: a cost modeling module for establishing a multi-dimensional cost calculation model by determining hierarchical storage costs, data access costs and migration operation costs based on cloud resource platform billing rules and price data; an index evaluation module for generating elastic billing evaluation indexes by analyzing cost influence factors of data migration using the multi-dimensional cost calculation model; 2. The cloud resource based data tiering migration service system as claimed in claim 1, wherein, a decision optimization module for constructing a cost-optimized migration decision function according to the elastic billing evaluation indexes, and determining a migration timing schedule. The cost decision unit is connected with the batch planning unit and the migration execution unit; The cost modeling module comprises: a data acquisition module for acquiring hierarchical storage billing data, data access billing data and migration operation billing data of the cloud resource platform through an API interface, and establishing a multi-dimensional billing data set; a cost analysis module for analyzing hierarchical storage costs, data access costs and migration operation costs based on the multi-dimensional billing data set, in combination with different storage level price coefficients and billing period information of the cloud resource platform; 3.The cloud resource based data tiering migration service system of claim 2, wherein, a model construction module for establishing a multi-dimensional cost calculation model according to the hierarchical storage costs, the data access costs and the migration operation costs. The analysis of the hierarchical storage costs, the data access costs and the migration operation costs comprises: extracting unit capacity prices, minimum storage duration requirements and early deletion penalty fees of different storage levels based on hierarchical storage billing data, and obtaining hierarchical storage costs in combination with actual occupied space and expected retention period of data blocks; calculating data access costs in combination with historical access patterns and access frequency statistical information according to read-write operation unit prices, data retrieval fees and bandwidth rates in data access billing data; 4. The cloud resource based data tiering migration service system as claimed in claim 2, wherein, analyzing billing trough periods and peak periods of different migration periods by using data transmission fees, computing resource occupation fees and cross-region additional fees in migration operation billing data, and determining migration operation costs. The index evaluation module comprises: a cost estimation module for determining current storage costs, access costs and operation costs of potential migration paths of each data block distributed in different storage levels based on the multi-dimensional cost calculation model and historical storage records of the cloud resource platform; a factor extraction module for calculating cost change rates, resource utilization efficiencies and billing period coefficients under different migration strategies according to the current storage costs, the access costs and the operation costs of the potential migration paths of each data block, and generating key cost influence factors; an index construction module for establishing elastic billing evaluation indexes including storage level sensitivity, access frequency threshold and migration timing critical value based on the key cost influence factors.

5. The cloud resource based data tiering migration service system as claimed in claim 2, wherein, The decision optimization module comprises: a function establishment module, configured to establish a migration target function based on the elastic billing evaluation index, with the objective of minimizing migration cost and in combination with access performance constraints; a timing identification module, configured to calculate cost-benefit scores of performing migration operations at different time points by using the migration target function, and identify a migration time window with the lowest cost; a schedule generation module, configured to determine target migration timing and priority ranking of each data block according to the cost-benefit scores and migration urgency, and establish a migration schedule table.

6. The cloud resource based data tiering migration service system as claimed in claim 1, wherein, The batch planning unit comprises: a demand analysis module, configured to analyze time distribution characteristics of target migration timing of each data block by using the migration timing schedule table, and construct a migration demand distribution matrix in combination with the elastic billing evaluation index; a constraint evaluation module, configured to analyze resource constraint influence coefficients according to real-time bandwidth states and storage hierarchy load conditions of the cloud resource platform, and dynamically adjust migration parameters based on preset threshold values; a batch generation module, configured to generate migration planning batches based on the migration demand distribution matrix and the resource constraint influence coefficients.

7. The cloud resource based data tiering migration service system as claimed in claim 6, wherein, The demand analysis module comprises: an information extraction module, configured to extract target migration timing and priority ranking information of each data block based on the migration timing schedule table; a feature identification module, configured to analyze time distribution characteristics of target migration timing of each data block, and identify migration demand density and resource competition degree within a billing period; a matrix construction module, configured to construct a migration demand distribution matrix in combination with the migration demand density, the resource competition degree, and storage hierarchy sensitivity and access frequency threshold values in the elastic billing evaluation index.

8. The cloud resource based data tiering migration service system as claimed in claim 6, wherein, The analysis of the resource constraint influence coefficients according to real-time bandwidth states and storage hierarchy load conditions of the cloud resource platform, and the dynamic adjustment of the migration parameters based on preset threshold values comprise: calculating the resource constraint influence coefficients; if the network bandwidth availability of the cloud resource platform is lower than a bandwidth safety threshold value, reducing the migration data volume within a unit time and correspondingly increasing the number of migration batches according to a bandwidth adjustment ratio; if the IOPS usage rate of a target storage hierarchy exceeds an upper load threshold value, adjusting the migration execution time to a time period in which the IOPS usage rate of the storage hierarchy is lower than a load suitable threshold value; if the number of access operations of a source storage hierarchy increases at a rate exceeding an access fluctuation threshold value within a monitoring period, delaying non-critical migration tasks to the next billing period and recalculating the resource constraint influence coefficients. 9.The cloud resource based data tiering migration service system of claim 8, wherein, The calculation of the resource constraint influence coefficients comprises: real-time acquisition of network bandwidth usage, storage hierarchy IOPS load level, and access operation frequency of the cloud resource platform through an API interface, and solving the resource constraint influence coefficients; an expression of the resource constraint influence coefficients is: ； In the formula, IC is a resource constraint impact coefficient; BW r is a current network bandwidth availability rate; BW max is a maximum network bandwidth availability rate; IOPS c is a target storage tier current IOPS usage rate; IOPS max is a target storage tier maximum IOPS threshold; Δ AO r is a source storage tier access operation quantity growth rate; AO max is a maximum allowed access operation growth rate; α 、 β 、 γ are weight coefficients of the network bandwidth factor, the IOPS load factor, and the access operation factor, respectively.

10. The cloud resource based data tiering migration service system as claimed in claim 6, wherein, The batch generation module comprises: a scheme formulation module, configured to determine migration priorities according to the migration demand distribution matrix, and establish a batch allocation scheme of each data block in combination with the resource constraint influence coefficients; a task merging module, configured to merge migration tasks within a same billing period in the batch allocation scheme based on billing period information of the cloud resource platform, so as to reduce migration operation cost. The planning output module is configured to generate a migration planning batch containing a batch sequence, an estimated execution time, and a resource requirement amount, and to synchronize the migration planning batch with a migration timing schedule in time.

Images