Dynamic Migration Method for Distributed Storage Replicas Based on Access Popularity Prediction

By using a distributed storage system based on access popularity prediction, and leveraging LSTM networks and multi-dimensional migration cost assessment, we can proactively optimize migration decisions before hotspots form, solving the problems of migration lag and inefficient migration in existing technologies, and improving system performance and resource utilization efficiency.

CN123114385APending Publication Date: 2026-11-17HEFEI XINLI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611621003.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing distributed storage systems, the passive response mode of replica migration methods causes migration to lag behind the formation of hotspots. Users experience performance degradation before the migration is completed, and the migration decision fails to effectively assess the trade-off between transmission overhead, service degradation costs, and the degree of load relief after migration, which may trigger inefficient migration.

Method used

By using a method based on access popularity prediction, the spatiotemporal characteristics of data sharding are predicted using an LSTM network. Combined with multi-dimensional migration cost evaluation, migration decisions are generated, including data transmission, service degradation, resource consumption, and migration frequency costs. The net benefit calculation of migration operations and the duration constraint of popularity are optimized to achieve closed-loop calibration of proactive migration decisions and migration effects.

Benefits of technology

It effectively avoids inefficient migration, reduces unnecessary migration operations, improves the accuracy of migration decisions and system performance, and ensures efficient resource optimization and allocation before hotspots form.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114385A_ABST
    Figure CN123114385A_ABST
Patent Text Reader

Abstract

This invention specifically relates to a dynamic migration method for distributed storage replicas based on access popularity prediction, belonging to the field of distributed storage technology. The method includes: selecting data shards whose predicted access popularity exceeds a hotspot determination threshold as candidate shards for migration; calculating the data transmission cost, service degradation cost, resource consumption cost, and migration frequency cost for each candidate migration operation; and inputting these values ​​into a cost evaluation model after normalization to generate the total migration cost. In this invention, the cost of candidate migration operations is quantified from four dimensions: data transmission time, service degradation, target node resource consumption, and migration frequency. After normalization to eliminate dimensional differences, the net benefit is calculated and a threshold is determined based on the expected migration benefit. This ensures that each migration operation is triggered only when the net benefit is positive and exceeds the threshold, avoiding inefficient migrations where the cost exceeds the benefit.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a method for dynamic migration of distributed storage replicas based on access popularity prediction. Background Technology

[0002] In a distributed storage system, when a data node carrying a data shard is accessed by a large number of users, that node becomes an access hotspot. Access hotspots lead to increased node response latency and a subsequent decrease in service quality.

[0003] Existing replica migration methods typically employ a passive response mode, meaning that migration operations are only triggered after a node load exceeds a threshold.

[0004] The passive response mode has two problems: First, the migration action lags behind the formation of hotspots, and users have already experienced performance degradation before the migration is completed; second, the migration decision is based solely on the current load status, without assessing the trade-off between the transmission overhead of the migration operation itself, the cost of service degradation, and the degree and duration of load relief after migration, which may trigger an inefficient migration where the cost outweighs the benefits. Summary of the Invention

[0005] The purpose of this invention is to propose a dynamic migration method for distributed storage replicas based on access popularity prediction in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A distributed storage replica dynamic migration method based on access popularity prediction includes: Historical access logs of each data segment are obtained, and time dimension features and spatial dimension features are extracted from the historical access logs respectively. The extracted time dimension access features and spatial dimension access features are concatenated into a spatiotemporal feature vector. The spatiotemporal feature vector sequence is input into an LSTM network to output the predicted access popularity and the predicted duration of popularity of each data segment in multiple future time windows. Data shards whose predicted access popularity exceeds the hotspot determination threshold are selected as candidate shards to be migrated. For each candidate migration operation, the data transmission cost, service degradation cost, resource consumption cost, and migration frequency cost are calculated separately. After normalization, the data is input into the cost evaluation model to generate the total migration cost. The expected migration benefit is calculated based on the expected load relief after migration and the predicted duration of the heat wave. The difference between the normalized expected migration benefit and the total migration value is taken as the net benefit. A migration decision is generated by combining the predicted duration of the heat wave and the net benefit.

[0007] Preferably, the time-dimensional access features include periodic, burst, and trend components of the access frequency. The periodic component is obtained by extracting the periodic frequency component through a Fourier transform of the access frequency time series. The burst component is obtained by detecting pulse segments in the access frequency time series that deviate from the periodic component by more than a burst threshold. The trend component is obtained by obtaining the slope value through a sliding window linear fitting of the access frequency time series. The spatial-dimensional access features include the distribution of access proportions for each network topology region to the data fragment. Before concatenating into the spatiotemporal feature vector, the periodic component, the burst component, and the trend component are respectively subjected to Z-score normalization.

[0008] Preferably, the LSTM network also outputs confidence intervals for the predicted access popularity of each data slice within each future time window. The confidence intervals are defined by adding or subtracting twice the standard deviation of the prediction error from the predicted access popularity of each time window as the upper and lower bounds, respectively. The standard deviation of the prediction error is obtained by statistically analyzing the prediction error sequence of the corresponding time window on the historical validation set.

[0009] Preferably, the calculation of the service degradation cost includes: obtaining the number of pending requests currently being processed by the source node. and the number of pending requests at the target node Calculate the expected proportion of I / O resources to be used during the migration. ,in This represents the currently available network bandwidth between the source node and the target node. Maximum I / O throughput capacity of the source node; service degradation cost .

[0010] Preferably, the calculation of the migration frequency cost value includes: obtaining the historical migration count of the candidate fragment to be migrated within a preset time window. and preset frequency threshold ,when Exceed At that time, the value of migration frequency ,in The preset frequency penalty coefficient; when Not exceeding hour, It is zero.

[0011] Preferably, the step of generating a migration decision by combining the predicted duration of popularity and the net benefit value includes: generating a no-migration decision when the predicted duration of popularity does not exceed a duration threshold; generating a migration decision when the predicted duration of popularity exceeds the duration threshold and the net benefit value exceeds a preset net benefit threshold; generating a delayed migration decision when the predicted duration of popularity exceeds the duration threshold and the net benefit value is positive but does not exceed the preset net benefit threshold; and generating a no-migration decision when the net benefit value does not exceed zero; wherein the value of the duration threshold is greater than the expected transmission time of the candidate migration operation.

[0012] Preferably, when the predicted access popularity values ​​of multiple data shards simultaneously exceed the hotspot determination threshold and there is access correlation among the multiple data shards, the net benefit value of the batch migration of the data shard group with access correlation is calculated as a whole, and compared with the sum of the net benefit values ​​of the individual migration of each data shard, and the scheme with the higher net benefit value is selected for execution; the access correlation is determined by calculating the cosine similarity between the access source distribution vectors of each data shard, and when the cosine similarity exceeds the correlation threshold, the two data shards are identified as having access correlation.

[0013] Preferably, the process further includes: performing coordinated scheduling on multiple candidate migration operations that generate migration decisions; calculating the scheduling priority of each migration operation, wherein the scheduling priority is the ratio of the net benefit value of the migration operation to the expected transmission time; generating a migration execution queue by arranging the scheduling priorities in descending order; traversing the migration execution queue to detect migration operations sharing the same network link; allocating migration operations sharing the same link to different execution time periods; and initiating data transmission for each migration operation in an online migration manner according to the order of the migration execution queue and the allocated execution time periods.

[0014] Preferably, the method further includes: collecting actual running data within a verification time window after the migration operation is completed, comparing the actual migration effect with the corresponding predicted value, and calculating the deviation value of each item; when the predicted deviation value of access popularity exceeds the prediction deviation threshold, using the actual access popularity data as a new training sample to perform incremental training on the LSTM network; when the deviation value of the total migration cost exceeds the cost deviation threshold, refitting the weight parameters of each dimension of the cost value in the cost evaluation model based on the actual cost data.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention quantifies the cost of candidate migration operations from four dimensions: data transmission time, service degradation, target node resource consumption, and migration frequency. After normalization to eliminate dimensional differences, it calculates the net benefit and determines the threshold with the expected migration benefit. This ensures that each migration operation is only triggered when the net benefit is positive and exceeds the threshold, thus avoiding inefficient migrations where the cost exceeds the benefit.

[0016] 2. This invention introduces the predicted duration of popularity as a decision constraint, eliminating invalid scenarios where migration is performed when the hot topic is about to naturally fade away, thus reducing unnecessary migration operations. Through effect verification and model calibration closed loop after migration, the parameters of the LSTM network and cost evaluation model are continuously updated with the accumulation of running data, and the accuracy of migration decisions gradually improves with running time. Attached Figure Description

[0017] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0019] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0020] Example 1 Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0021] In this embodiment, it includes: In distributed storage systems, when a data node carrying a data shard is accessed by a large number of users, that node becomes an access hotspot, leading to increased response latency and decreased service quality. Existing replica migration methods typically employ a passive response model, triggering migration only after a node's load exceeds a threshold. This passive response model has two problems: first, migration lags behind the formation of the hotspot, meaning users experience performance degradation before migration is complete; second, migration decisions are based solely on the current load state, failing to assess the trade-off between the migration operation's transmission overhead, service degradation costs, and the degree and duration of load mitigation after migration, potentially triggering inefficient migrations where the costs outweigh the benefits.

[0022] This embodiment provides a dynamic migration method for distributed storage replicas based on access popularity prediction. By predicting the spatiotemporal evolution of data shard access popularity and combining it with multi-dimensional real-time evaluation of migration costs, proactive migration decisions are made before hot spots form, and the prediction and evaluation process is calibrated in a closed loop through migration effect verification.

[0023] The method described in this embodiment runs on the management node of a distributed storage system. The management node maintains communication connections with each storage node and is able to collect access logs, load status, and network bandwidth information from each storage node.

[0024] A distributed storage replica proactive migration method according to an embodiment of the present invention includes the following steps: Step 1: Obtain historical access logs for each data shard and generate spatiotemporal evolution prediction results for access popularity;

[0025] Historical access logs for each data shard are collected from each storage node of the distributed storage system. The historical access logs include the timestamp of each access record, the network topology location from which the access originated, and the identifier of the accessed data shard.

[0026] Time-dimensional features are extracted from historical access logs to obtain the time-dimensional access features for each data shard. These features include periodic, burst, and trend components of access frequency. The periodic component is obtained by performing a Fourier transform on the access frequency time series to extract daily and weekly frequency components. The burst component is obtained by detecting pulse segments in the access frequency time series that deviate from the periodic component by more than a burst threshold. The burst threshold is a preset access frequency deviation, with the same unit as the access frequency, used to distinguish between normal fluctuations and sudden access pulses. The trend component is obtained by performing a sliding window linear fitting on the access frequency time series to obtain the slope value, representing the direction of increase or decrease in access volume over time.

[0027] Spatial dimension features are extracted from historical access logs to obtain the spatial dimension access features for each data shard. These features include the distribution of access percentages for each data shard across different network topology regions.

[0028] Before concatenating the temporal and spatial access features into a spatiotemporal feature vector, data preprocessing is performed on each component. The periodic, bursty, and trend components of access frequency have different dimensions (frequency, frequency deviation, and frequency slope, respectively), and the access proportion distribution for each network topology region is already a dimensionless proportional value. Z-score standardization is applied to the periodic, bursty, and trend components to eliminate the impact of dimensional differences on subsequent model inputs; the spatial access features are already proportional values ​​and do not require additional normalization.

[0029] The standardized temporal and spatial access features are concatenated into a spatiotemporal feature vector. The spatiotemporal feature vector sequences of each data segment are then input into an LSTM network in time-step order. The network outputs the predicted access frequency for each data segment over multiple future time windows, along with its corresponding confidence interval. The predicted access frequency represents the expected access frequency of that data segment within the corresponding time window. The confidence interval is obtained by statistically analyzing the prediction error distribution between the predicted and actual access frequency values ​​for each time window on the historical validation set. The upper and lower bounds of the confidence interval are defined as the predicted access frequency value for each time window plus or minus twice the standard deviation of the prediction error, respectively, representing the range of uncertainty in the prediction. The standard deviation of the prediction error is obtained statistically from the prediction error sequence for the corresponding time window on the historical validation set. The LSTM network is trained using the actual access frequency values ​​of known time windows from historical access logs as supervision labels.

[0030] It should be noted that the above-mentioned visit popularity prediction also includes popularity duration prediction. Popularity duration prediction refers to the duration from the start time window when the predicted visit popularity value exceeds the hotspot determination threshold to the end time window when the predicted popularity value falls back below the hotspot determination threshold. Popularity duration prediction is obtained by traversing the predicted value sequence of future time windows, determining the number of time windows that consecutively exceed the hotspot determination threshold, and multiplying this number by the duration of a single time window.

[0031] Step 2: Conduct a multi-dimensional cost evaluation of candidate migration operations to generate the total migration value; Based on the access popularity prediction results, data shards whose predicted access popularity values ​​exceed the hotspot determination threshold are selected as candidate shards for migration. For each candidate shard to be migrated, candidate target nodes that can receive the candidate shard are determined, forming a set of candidate migration operations. Each candidate migration operation is represented by a triplet of source node, target node, and shard to be migrated.

[0032] For each candidate migration operation, calculate the cost across the following four dimensions. Step 2 may specifically include: Step 201: Calculate data transmission cost. Obtain the data volume of the shard to be migrated and the current available network bandwidth between the source node and the target node, and calculate the data transmission cost. The data transmission cost is the ratio of the data volume of the shard to be migrated to the available network bandwidth, representing the estimated transmission time of the migration operation, expressed in time units.

[0033] Step 202, calculate the service degradation cost. Obtain the current request queue depths of the source node and the target node, denoted as follows: and ,in This represents the number of requests currently pending on the source node. This represents the number of pending requests currently being processed on the target node; both are dimensionless request counts. The estimated percentage of I / O resources to be used during the migration. Based on the current available network bandwidth between the source node and the target node Maximum I / O throughput capacity of the source node The ratio is calculated to obtain, that is ,in and The units are all the amount of data that can be transmitted per unit of time. This is a dimensionless proportional value. It represents the expected request-response latency increment of the source node. by and The product representation, i.e. The larger the value, the greater the impact of the migration on the service capacity of the source node; the expected request response latency increment of the target node. by and The product representation, i.e. The cost of service downgrade for and The sum of Its dimensions are the same as , The same, is a dimensionless weighted sum of request counts.

[0034] Step 203: Calculate the resource consumption cost of the target node. Obtain the current storage space utilization and bandwidth utilization of the target node, and calculate the increase in storage space utilization of the target node after the migration is completed. and bandwidth utilization increment Both are dimensionless proportional values, ranging from 0 to 1. Resource occupation cost for and The weighted sum, i.e. ,in and The preset weighting coefficients, It reflects the relative impact of storage space usage on the resource status of the target node. It reflects the relative impact of bandwidth usage on the resource status of the target node.

[0035] Step 204: Calculate migration frequency cost. Query the historical migration counts of the candidate shard within a preset time window. ,in The migration count is a dimensionless value, and a preset frequency threshold is obtained. ,in The preset upper limit for the number of migrations, with units of... Same. When Exceed At that time, the value of migration frequency according to Calculation, where A preset frequency penalty coefficient is used to quantify the incremental cost of each additional migration exceeding the frequency threshold; when Not exceeding hour, It is zero.

[0036] Before inputting the cost values ​​of the four dimensions into the cost evaluation model, data preprocessing is performed on the cost values ​​of each dimension. The unit of data transmission cost is time unit, the unit of service degradation cost is a dimensionless request count weighted value, the unit of resource consumption cost is a dimensionless proportion value, and the unit of migration frequency cost is a dimensionless count value. The units of each dimension are inconsistent. The cost values ​​of the four dimensions are respectively processed by mean normalization based on range, and after mapping the cost values ​​of each dimension to the same numerical range, they are combined into a four-dimensional input vector.

[0037] The normalized four-dimensional vector is input into the cost assessment model, which outputs the total migration cost value. The cost assessment model is a linear regression model. Its input consists of a four-dimensional vector representing the normalized data transmission cost value, service degradation cost value, resource consumption cost value, and migration frequency cost value. The output is a scalar value of the total migration cost, which is a weighted linear combination of the four cost dimensions. The cost assessment model uses the actual comprehensive cost of historical migration operations as a supervision label and fits the weight parameters of each cost dimension using the least squares method.

[0038] Step 3: Based on the predicted access popularity and the total value of migration, weigh the costs and benefits to generate a migration decision;

[0039] For each candidate migration operation, calculate the expected migration benefit value. The expected migration benefit value is based on two factors: the expected load reduction after migration and the duration of the load reduction. The expected load reduction after migration is the difference between the predicted access popularity value of the source node before migration and the predicted access popularity value after migration, expressed in access frequency units. The duration of the load reduction is the predicted duration of popularity obtained in step 1, expressed in time units. The expected migration benefit value is the product of the expected load reduction after migration and the duration of the load reduction, expressed as the product of access frequency and time, representing the total number of access requests cumulatively reduced during the hotspot period.

[0040] It should be noted that the total migration value is output by the cost assessment model and is a normalized dimensionless scalar, which has different dimensions from the expected migration benefit value and cannot be directly subtracted. To ensure the comparability of the net benefit value calculation, the expected migration benefit value is normalized to the mean based on the range before calculating the net benefit value, mapping the expected migration benefit value to the same numerical range as the total migration value.

[0041] Calculate the net income value. The net income value is the difference between the normalized expected migration income value and the total migration agency value.

[0042] Based on the net profit value and the predicted duration of popularity, a ternary migration decision is generated. If the predicted duration of popularity does not exceed a duration threshold, a no-migration decision is generated. If the predicted duration of popularity exceeds the duration threshold, the net profit value is further evaluated: if the net profit value exceeds a preset net profit threshold, a migration decision is generated; if the net profit value is positive but does not exceed the preset net profit threshold, a delayed migration decision is generated, and the time point of the next evaluation is recorded; if the net profit value does not exceed zero, a no-migration decision is generated.

[0043] In this embodiment, to avoid performing invalid migrations when a hotspot is about to naturally fade away, the duration threshold is set to be greater than the expected transmission time of the candidate migration operation. When the predicted duration of the hotspot is less than the expected transmission time, a no-migration decision is generated even if the net benefit is positive. This constraint ensures that the hotspot will not naturally fade away before the migration operation is completed.

[0044] In this embodiment, when the prediction results show that the access popularity of multiple data shards will increase simultaneously and there is an access correlation between these data shards, a batch migration decision is performed on these correlated data shards. Access correlation is determined by calculating the cosine similarity between the access source distributions of each data shard. When the cosine similarity exceeds the correlation threshold, the two data shards are considered to have an access correlation. The access source distribution vectors of each data shard are constructed with the entire network topology region as the dimension, and each dimension represents the access proportion of the corresponding region. Therefore, the access source distribution vectors of any two data shards have the same dimension, and cosine similarity calculation can be performed directly. For data shard groups with access correlation, the net benefit value of the batch migration is calculated as a whole for the data shard group and compared with the sum of the net benefit values ​​of the individual migrations of each data shard. The scheme with the higher net benefit value is selected for execution.

[0045] Step 4: Perform cooperative scheduling and parallel migration on the candidate migration operations that generate migration decisions; When multiple candidate migration operations simultaneously generate migration decisions, coordinated scheduling is performed on these migration operations. Step 4 may specifically include: Step 401: Calculate the scheduling priority of each migration operation. The scheduling priority is the ratio of the net benefit value of the migration operation to the expected transmission time. Arrange the migration operations in descending order of scheduling priority to generate a migration execution queue.

[0046] Step 402: Perform bandwidth contention detection. Traverse all migration operations in the migration execution queue to detect if two or more migration operations share the same network link. When a shared link is detected, allocate the migration operations on the shared link to different execution time slots, ensuring that the same network link carries only the data transmission of one migration operation at a time.

[0047] Step 403: Initiate data transmission for each migration operation according to the order of the migration execution queue and the allocated execution time period. Data transmission adopts an online migration method, maintaining the read and write service availability of the data shard on the source node during the transmission process.

[0048] In this embodiment of the application, in order to prevent the migration operation from causing the target node to overload, the load value of each target node is monitored in real time during the migration process. When the load value of a target node is detected to exceed the target node's load limit, the migration operation targeting that target node is paused, the migration operation is added back to the migration execution queue, and steps 2 and 3 are re-executed to evaluate whether to replace the target node.

[0049] Step 5: Verify the migration effect and calibrate the LSTM network and cost evaluation model; Within the verification time window following the completion of the migration operation, actual runtime data is collected to obtain the actual migration effect. The actual migration effect includes the following three items: the actual load reduction of the source node, the actual migration time and actual resource consumption, and the actual change in user access latency.

[0050] The actual migration effect values ​​are compared with the corresponding predicted values, and the deviation values ​​for each item are calculated. The deviation value is the ratio of the absolute value of the difference between the actual value and the predicted value to the predicted value.

[0051] Model calibration based on deviation values: For LSTM networks, when the predicted access popularity deviation value of a certain data segment exceeds the prediction deviation threshold, the actual access popularity data of that data segment is used as a new training sample to perform incremental training on the LSTM network and update the model parameters of the LSTM network; for cost evaluation models, when the deviation value of the total transfer cost exceeds the cost deviation threshold, the weight parameters of each dimension of the cost evaluation model are refitted based on the actual cost data.

[0052] It should be noted that the above calibration process is performed in a continuous loop. An effect verification and bias calculation are triggered after each migration operation. When the accumulated number of new training samples reaches the batch calibration threshold, a batch calibration is performed on the LSTM network and the cost evaluation model. The batch calibration uses all new training samples from the most recent calibration period.

[0053] Example 2 A video-on-demand platform's distributed storage system consists of multiple storage nodes responsible for storing and providing data shard services for popular film and television content. Historically, around the time of a popular variety show's premiere in a certain year, the platform has repeatedly experienced instances of concentrated access to specific data shards, causing node hotspots. The management node continuously collects access logs, load status, and network bandwidth information from each storage node. Currently, data shard "Shard A" is stored on node 1. Historical data shows that its access volume experiences periodic peaks every Friday evening, with a recent upward trend. The management node predicts the future access volume of Shard A and assesses whether its replicas need to be migrated to node 2 or node 3 before a hotspot forms.

[0054] The execution process of step 1; The management node collects historical access logs from Node 1 for shard A, extracting access features in both time and spatial dimensions. In the time dimension, Fourier transform extracts daily and weekly periodic components, detecting two sudden spikes within the past two weeks. Sliding window linear fitting shows the trend component has a positive slope, indicating an overall increasing access volume. Spatially, the distribution of access sources shows the East China region has the highest proportion. After Z-score standardization of the periodic, sudden, and trend components, these are concatenated with the spatial access proportions to form a spatiotemporal feature vector. This vector is input into an LSTM network, outputting predicted access popularity values ​​and confidence intervals for the next four time windows (each window lasting 2 hours). The hotspot determination threshold is set to 1200 requests per second, as shown in Table 1 below.

[0055] Table 1. Prediction Results of Visit Popularity for Segment A

[0056] If the predicted popularity value exceeds the hotspot determination threshold continuously from window 2 to window 4, the predicted duration of the popularity is 6 hours (3 windows multiplied by 2 hours / window).

[0057] The execution process of step 2; Shard A is selected as a candidate shard for migration. The candidate target nodes are nodes 2 and 3, resulting in two candidate migration operations: operation A (node ​​1 → node 2) and operation B (node ​​1 → node 3). Taking operation A as an example, the cost is calculated for each of the four dimensions.

[0058] Data transmission cost: The data volume of fragment A is 480GB, the current available bandwidth between node 1 and node 2 is 800Mbps, and the data transmission cost (estimated transmission time) is approximately 4800 seconds (i.e. 80 minutes).

[0059] Service degradation cost: Current request queue depth on node 1 The current request queue depth for node 2 is 85. The maximum I / O throughput capacity of node 1 is 42. Corresponding to 800Mbps, currently available bandwidth If it is 800Mbps, then The source node is expected to experience an increase in response latency. The target node is expected to increase the response latency. Service downgrade cost .

[0060] Resource consumption cost: Increment in storage space utilization of Node 2 after migration. The bandwidth utilization increment is 0.08. The preset weight is 0.12. , ,but .

[0061] Migration frequency cost: The number of times fragment A migrates historically within a preset time window (last 7 days). The preset frequency threshold is 2. It is 3. Not exceeding Therefore .

[0062] After performing mean normalization based on the range on the cost values ​​of the four dimensions, a four-dimensional input vector is formed, which is input into the cost evaluation model, and the total migration cost value is output, as shown in Table 2 below.

[0063] Table 2 Calculation results of candidate migration operation costs

[0064] The available bandwidth between node 1 and node 3 is low (640Mbps), and the bandwidth utilization of node 3 is high. As a result, operation B is more expensive than operation A in both data transmission cost and resource consumption cost. Ultimately, the total migration value of operation A is lower than that of operation B.

[0065] The execution process of step 3; Calculate the expected migration benefit for operation A. After migration, the predicted access frequency of the source node (node ​​1) will decrease from a peak of 1580 visits / second to approximately 420 visits / second (the reduction in access requests for shard A on node 1 after the shard A replica is migrated to node 2). The expected load relief is 1160 visits / second, and the duration of load relief is the predicted duration of the high-frequency access obtained in step 1, which is 6 hours (21600 seconds). The expected migration benefit is... The unit is times per second. The expected migration gain is normalized to the mean based on the range, mapping it to the same numerical range as the total migration value. After normalization, the expected migration gain is 0.79.

[0066] The net profit of operation A is The net profit of operation B is .

[0067] The predicted duration of the heatwave is 6 hours, and the duration threshold is set to 2 hours (greater than the estimated transmission time of 80 minutes for operation A). 6 hours exceeds the duration threshold. The net revenue threshold is set to 0.20, as shown in Table 3 below.

[0068] Table 3 Decision Results of Candidate Migration Operations

[0069] Both operations A and B meet the conditions that their net profit value exceeds the net profit threshold and their popularity duration exceeds the duration threshold, so both generate a "migration" decision. Operation A has a higher net profit value, so it proceeds to the collaborative scheduling process in step 4.

[0070] It should be noted that the system also detected that the predicted access popularity of shard B would exceed the hotspot determination threshold within the same time window. The cosine similarity between the access source distribution vectors of shard A and shard B was calculated, yielding a result of 0.87, exceeding the association threshold of 0.80, thus determining that the two have an access association. The system calculated the net benefit value of the batch migration for the data shard group composed of shard A and shard B, and compared it with the sum of the net benefit values ​​of the individual migrations of each shard. The final batch migration plan showed a higher net benefit value, thus determining to perform a batch migration of the two shards to node 2.

[0071] The execution process of step 4; Both the migration operations of fragment A and fragment B generate a "migration" decision and enter the collaborative scheduling process.

[0072] Scheduling priority calculation: The net benefit of fragment A's migration operation is 0.38, the estimated transmission time is 4800 seconds, and the scheduling priority is [missing information]. The net benefit of the fragment B migration operation is 0.31, the estimated transmission time is 3600 seconds, and the scheduling priority is [missing information]. Slices are arranged in descending order of scheduling priority, with slice B having a higher priority than slice A and thus placed at the front of the execution queue.

[0073] Bandwidth contention detection: Both migration operations target node 2 and share the same network link from node 1 to node 2, resulting in bandwidth contention. The two operations are assigned to different execution periods: fragment B is executed in period 1, and fragment A is executed in period 2 (period 2 starts after fragment B transmission is complete), ensuring that the same link carries only one migration operation at any given time, as shown in Table 4 below.

[0074] Table 4. Results of Coordinated Scheduling of Migration Operations

[0075] Both migration operations were performed online, and the read / write services for the corresponding data shards on the source node remained available during the transfer. The load value of node 2 was monitored in real time during the migration, and the load limit protection was not triggered.

[0076] The execution process of step 5; Within the verification time window (2 hours) after the migration of fragment A is completed, the management node collects actual running data, obtains the actual value of the migration effect, and compares it with the predicted values ​​of steps 1 and 2 to calculate various deviation values, as shown in Table 5 below.

[0077] Table 5. Verification results of migration effect

[0078] The deviation of the user access latency change is 0.20, exceeding the prediction deviation threshold of 0.15, triggering the incremental training calibration process of the LSTM network. The actual access popularity data of segment A is added to the sample pool as new training samples. When the number of new samples accumulated in the sample pool reaches the batch calibration threshold, batch incremental training is performed on the LSTM network to update the model parameters. Since the total transfer cost deviation does not exceed the cost deviation threshold, the cost evaluation model weight parameters are not refitted in this round.

[0079] Throughout the implementation process, data starts from historical access logs, undergoes spatiotemporal feature extraction and LSTM network prediction to generate predicted access popularity and duration of popularity for each time window. The prediction results drive the selection of candidate migration operations. The four-dimensional cost value, after normalization, is input into the cost evaluation model to generate the total migration cost value. The net profit value is obtained by subtracting the normalized expected migration benefit value from the total migration cost value, and combined with a threshold of the predicted duration of popularity to generate a ternary migration decision. Operations that obtain a "migration" decision enter collaborative scheduling, are prioritized, and executed in staggered shifts. Actual operational data after migration completion is fed back to the deviation calculation stage, triggering closed-loop calibration of model parameters. The data connection between each step ensures the logical coherence of the five stages: prediction, evaluation, decision-making, execution, and calibration.

[0080] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0081] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0082] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for dynamic migration of distributed storage replicas based on access popularity prediction, characterized in that, include: Historical access logs of each data segment are obtained, and time dimension features and spatial dimension features are extracted from the historical access logs respectively. The extracted time dimension access features and spatial dimension access features are concatenated into a spatiotemporal feature vector. The spatiotemporal feature vector sequence is input into an LSTM network to output the predicted access popularity and the predicted duration of popularity of each data segment in multiple future time windows. Data shards whose predicted access popularity exceeds the hotspot determination threshold are selected as candidate shards to be migrated. For each candidate migration operation, the data transmission cost, service degradation cost, resource consumption cost, and migration frequency cost are calculated separately. After normalization, the data is input into the cost evaluation model to generate the total migration cost. The expected migration benefit is calculated based on the expected load relief after migration and the predicted duration of the heat wave. The difference between the normalized expected migration benefit and the total migration value is taken as the net benefit. A migration decision is generated by combining the predicted duration of the heat wave and the net benefit.

2. The method for dynamic migration of distributed storage replicas based on access popularity prediction according to claim 1, characterized in that, The time-dimensional access features include periodic, burst, and trend components of the access frequency. The periodic component is obtained by extracting the periodic frequency component through Fourier transform of the access frequency time series. The burst component is obtained by detecting pulse segments in the access frequency time series that deviate from the periodic component by more than a burst threshold. The trend component is obtained by obtaining the slope value through sliding window linear fitting of the access frequency time series. The spatial-dimensional access features include the distribution of access proportion of each network topology region to the data fragment. Before concatenating into the spatiotemporal feature vector, the periodic component, the burst component, and the trend component are respectively subjected to Z-score normalization.

3. The distributed storage replica dynamic migration method based on access popularity prediction according to claim 1, characterized in that, The LSTM network also outputs confidence intervals for the predicted access popularity of each data slice within each future time window. The confidence intervals are defined by adding or subtracting twice the standard deviation of the prediction error from the predicted access popularity of each time window as the upper and lower bounds, respectively. The standard deviation of the prediction error is obtained by statistically analyzing the prediction error sequence of the corresponding time window on the historical validation set.

4. The distributed storage replica dynamic migration method based on access popularity prediction according to claim 1, characterized in that, The calculation of the service degradation cost includes: obtaining the number of pending requests currently being processed by the source node. and the number of pending requests at the target node Calculate the expected proportion of I / O resources to be used during the migration. ,in This represents the currently available network bandwidth between the source node and the target node. Maximum I / O throughput capacity of the source node; service degradation cost .

5. The method for dynamic migration of distributed storage replicas based on access popularity prediction according to claim 1, characterized in that, The calculation of the migration frequency cost includes: obtaining the historical migration count of the candidate fragment to be migrated within a preset time window. and preset frequency threshold ,when Exceed At that time, the value of migration frequency ,in The preset frequency penalty coefficient; when Not exceeding hour, It is zero.

6. The method for dynamic migration of distributed storage replicas based on access popularity prediction according to claim 1, characterized in that, The process of generating a migration decision by combining the predicted duration of popularity and the net benefit value includes: generating a no-migration decision when the predicted duration of popularity does not exceed a duration threshold; generating a migration decision when the predicted duration of popularity exceeds the duration threshold and the net benefit value exceeds a preset net benefit threshold; generating a delayed migration decision when the predicted duration of popularity exceeds the duration threshold and the net benefit value is positive but does not exceed the preset net benefit threshold; and generating a no-migration decision when the net benefit value does not exceed zero. The duration threshold is greater than the expected transmission time of the candidate migration operation.

7. The distributed storage replica dynamic migration method based on access popularity prediction according to claim 1, characterized in that, When the predicted access popularity values ​​of multiple data shards simultaneously exceed the hotspot determination threshold and there is access correlation among the multiple data shards, the net benefit value of the batch migration of the data shard group with access correlation is calculated as a whole, and compared with the sum of the net benefit values ​​of the individual migration of each data shard. The scheme with the higher net benefit value is selected for execution. The access correlation is determined by calculating the cosine similarity between the access source distribution vectors of each data shard. When the cosine similarity exceeds the correlation threshold, the two data shards are identified as having access correlation.

8. The distributed storage replica dynamic migration method based on access popularity prediction according to claim 1, characterized in that, Also includes: Cooperative scheduling is performed on multiple candidate migration operations that generate migration decisions. The scheduling priority of each migration operation is calculated. The scheduling priority is the ratio of the net benefit value of the migration operation to the expected transmission time. The migration execution queue is generated by arranging the scheduling priorities in descending order. The migration execution queue is traversed to detect migration operations that share the same network link, and the migration operations that share the link are allocated to different execution time periods; Data transmission for each migration operation is initiated online, following the order of the migration execution queue and the allocated execution time period.

9. The method for dynamic migration of distributed storage replicas based on access popularity prediction according to claim 1, characterized in that, Also includes: During the verification time window after the migration operation is completed, actual running data is collected, the actual migration effect is compared with the corresponding predicted value, and the deviation value of each item is calculated. When the prediction deviation value of access popularity exceeds the prediction deviation threshold, the actual access popularity data is used as a new training sample to perform incremental training on the LSTM network. When the deviation of the total migration cost exceeds the cost deviation threshold, the weight parameters of each dimension of the cost value in the cost evaluation model are refitted based on the actual cost data.