Data migration method and device, equipment and medium

Abstract

The invention belongs to the field of big data, and relates to a data migration method, which comprises the following steps: obtaining storage cluster pressure test data, and constructing a performance model, the pressure test data being obtained by testing under different loads. And acquiring real-time monitoring data, and evaluating the current pressure state in combination with the performance model. And if the threshold value is exceeded, extracting a historical data change rule, and generating a load prediction model. And calculating a target migration concurrency number based on the real-time performance index and the prediction model. And adjusting the current data migration concurrence number and the bandwidth limiting parameter to target values. And finally, carrying out data migration of the storage cluster according to a preset protocol and the target parameter. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the pressure measurement data and the real-time monitoring data can be stored in a block chain. The data migration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data migration method, apparatus, device and medium. Background Art

[0002] In current data storage systems, disaster recovery migration, a crucial means of ensuring data security and business continuity, faces numerous challenges in its implementation. In particular, during the migration process, the number of concurrent migrations and traffic width (i.e., bandwidth) are often set based on preset values ​​and lack the ability to dynamically adjust. When a cluster experiences a sudden surge in concurrent read and write requests, continuing to perform disaster recovery migration according to the original migration parameters will result in a dramatic increase in the load on the underlying storage cluster. This situation can easily lead to a continuous deterioration in the underlying cluster's performance, manifesting as slow request timeouts and system anomalies. In severe cases, this can directly impact normal read and write operations, degrading the user experience.

[0003] What’s more serious is that the current monitoring mechanism can often only issue an alarm after a problem occurs, but it is difficult to effectively identify the problem before it occurs. This means that in order to ensure that real-time disaster recovery does not affect the normal service of the cluster, it is necessary to rely on manual or programmatic capture of service anomaly feedback before shutting down the disaster recovery migration, which undoubtedly increases the risk and uncertainty of the system. In addition, with the continuous growth of data volume, the migration efficiency of traditional disaster recovery migration systems is also severely restricted by storage pressure and load capacity when facing large-scale data migration. In the existing technology, the migration process lacks a real-time monitoring and adjustment mechanism for the storage status, which makes the system seem powerless when faced with sudden load changes. In summary, there is an urgent need for a method that can dynamically adjust migration parameters in the face of sudden load changes to improve data migration efficiency. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a data migration method, apparatus, device and medium to solve the existing problem that it is difficult to dynamically adjust migration parameters when facing sudden load changes and it is impossible to improve data migration efficiency.

[0005] To solve the above technical problems, the present invention provides a data migration method using the following technical solutions:

[0006] Obtain stress test data of the storage cluster and build a performance model of the storage cluster based on the stress test data. The stress test data is obtained by stress testing the storage cluster under different load conditions; obtain real-time monitoring data of the storage cluster and use the performance model to evaluate and analyze the real-time monitoring data to obtain the current stress state value of the storage cluster; if the stress state value exceeds the preset stress threshold, extract the load change pattern from the historical data of the storage cluster to generate a load prediction model; obtain real-time performance indicators of the storage cluster and use the load prediction model to calculate the target migration concurrency of the storage cluster based on the real-time performance indicators; adjust the current data migration concurrency and bandwidth limit parameters of the storage cluster based on the target migration concurrency to obtain the target data migration concurrency corresponding to the data migration concurrency and the target bandwidth limit parameters corresponding to the bandwidth display parameters; use the preset data transmission protocol to migrate the data of the storage cluster according to the target data migration concurrency and target bandwidth limit parameters.

[0007] Furthermore, the steps for obtaining stress test data of the storage cluster include:

[0008] Obtain stress test conditions and test parameters for the storage cluster under different load conditions; perform stress tests on the storage cluster based on the stress test conditions and test parameters under different load conditions to generate performance data for the storage cluster under different load conditions; use a preset machine learning algorithm to deeply mine the performance data to obtain stress test data for the storage cluster.

[0009] Furthermore, the steps of obtaining real-time monitoring data of the storage cluster specifically include:

[0010] Obtain the monitoring rules and monitoring thresholds of the storage cluster; based on the monitoring rules, monitor the storage cluster in real time to obtain various monitoring indicator data of the storage cluster; compare the monitoring indicator data with the monitoring threshold to determine whether there is any abnormality in the storage cluster; if the monitoring indicator data exceeds the monitoring threshold, the monitoring indicator data is used as the real-time monitoring data of the storage cluster.

[0011] Furthermore, the steps of extracting load variation patterns from the historical data of the storage cluster and generating a load prediction model specifically include:

[0012] Obtain historical data of the storage cluster, sample the historical data according to the preset time granularity, and obtain a load change data series; perform feature extraction on the load change data series to obtain key features that reflect the periodicity and trend of the load change data series; and establish a load forecasting model based on the key features using a time series analysis algorithm.

[0013] Furthermore, based on real-time performance indicators and using a load prediction model, the target number of concurrent migrations for the storage cluster is calculated. Specifically, the steps include:

[0014] The real-time performance indicators are input into the load prediction model to obtain the load change trend prediction results of the storage cluster within the target time. Based on the load change trend prediction results, the target concurrency number of migrations of the storage cluster is calculated.

[0015] Furthermore, before migrating the data of the storage cluster using a preset data transmission protocol according to the target data migration concurrency number and the target bandwidth limit parameters, the method further includes:

[0016] Build a visual interface; visualize the real-time progress and status information of data migration to obtain a data migration visual interface.

[0017] Furthermore, after migrating the data of the storage cluster using a preset data transmission protocol according to the target data migration concurrency number and the target bandwidth limit parameter, the method further includes:

[0018] Obtain breakpoint location information during the data migration process; determine whether the breakpoint location information is empty; if the breakpoint location information is not empty, use a preset verification algorithm to verify the integrity of the migrated data and obtain a verification result; if the verification result shows that the migrated data is incomplete, obtain the corresponding target breakpoint location information from the breakpoint location information; and re-initiate data migration from the breakpoint indicated by the target breakpoint location information.

[0019] In order to solve the above technical problems, the present application also provides a data migration device, which adopts the following technical solutions:

[0020] The first acquisition module is used to obtain stress test data of the storage cluster and build a performance model of the storage cluster based on the stress test data. The stress test data is obtained by stress testing the storage cluster under different load conditions;

[0021] The second acquisition module is used to obtain real-time monitoring data of the storage cluster, use the performance model to evaluate and analyze the real-time monitoring data, and obtain the current pressure status value of the storage cluster;

[0022] An extraction module is used to extract load variation patterns from the historical data of the storage cluster and generate a load prediction model if the pressure state value exceeds a preset pressure threshold;

[0023] The calculation module is used to obtain the real-time performance indicators of the storage cluster and calculate the target concurrency of migrations of the storage cluster based on the real-time performance indicators and the load prediction model.

[0024] An adjustment module is used to adjust the current data migration concurrency and bandwidth limit parameters of the storage cluster based on the target migration concurrency, and obtain the target data migration concurrency corresponding to the data migration concurrency and the target bandwidth limit parameters corresponding to the bandwidth display parameters;

[0025] The data migration module is used to migrate the data of the storage cluster using a preset data transmission protocol according to the target data migration concurrency number and target bandwidth limit parameters.

[0026] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above-mentioned data migration method are implemented.

[0027] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable at least one processor to perform the steps of the above-mentioned data migration method.

[0028] Compared with the prior art, the embodiments of the present application have the following main beneficial effects: by addressing the problems of fixed migration parameters, lagging monitoring mechanisms and low migration efficiency in the existing disaster recovery migration process, an innovative solution is proposed. By obtaining the stress test data of the storage cluster to build a performance model and monitoring the cluster status in real time, the current pressure state can be accurately assessed, so that when the pressure exceeds the preset threshold, the load change pattern can be extracted from the historical data in a timely manner to generate a load prediction model. The load prediction model, combined with real-time performance indicators, can dynamically calculate the target migration concurrency, and then adjust the data migration concurrency and bandwidth limit parameters to achieve intelligent dynamic adjustment of migration parameters. This solution can not only effectively respond to sudden load changes, avoid the continuous deterioration of the underlying cluster performance, reduce the occurrence of slow request timeouts and system anomalies, ensure normal user read and write operations, and improve user experience, but also through real-time monitoring and adjustment mechanisms, it can effectively identify problems before they occur, reducing system risks and uncertainties. In addition, when facing large-scale data migration, this solution can significantly improve migration efficiency, reduce storage pressure and load capacity constraints, and ensure that the disaster recovery migration process is smoother and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 This is a flowchart of a data migration method provided by this application;

[0032] Figure 3 This is a structural diagram of a data migration device provided by this application;

[0033] Figure 4 This is a structural diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0035] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0036] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0038] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0039] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0040] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0041] It should be noted that the data migration method provided in the embodiment of the present application is generally executed by a server, and accordingly, the data migration device is generally set in the server.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0043] Continue to refer Figure 2 , shows a flow chart of an embodiment of a data migration method according to the present application. The data migration method comprises the following steps:

[0044] Step S201 : Obtain stress test data of the storage cluster, and build a performance model of the storage cluster based on the stress test data. The stress test data is obtained by performing stress tests on the storage cluster under different load conditions.

[0045] In this embodiment, the data migration method is executed on the electronic device (eg Figure 1 The server shown in the figure can obtain stress test data through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other wireless connection methods currently known or to be developed in the future.

[0046] A storage cluster is a cluster system consisting of multiple storage devices connected via a network. It can be used to implement distributed data storage and management, improving data storage reliability, scalability, and performance.

[0047] Stress testing data refers to the performance data of the storage cluster obtained through stress testing under different load conditions. Stress testing data may include response time, throughput, resource utilization, etc. Stress testing data can be used to assess the performance bottlenecks and load-bearing capacity of the storage cluster.

[0048] The performance model is a model built based on stress test data and used to simulate and predict the performance of the storage cluster under different load conditions.

[0049] In one embodiment, the storage cluster can be stress-tested through a preset performance evaluation and analysis module to obtain stress test data of the storage cluster. The performance evaluation and analysis module is a software system module that generates stress test data, which can perform performance testing and analysis on the storage cluster under different load conditions based on preset test conditions and parameters. The performance evaluation and analysis module is not limited to stress testing, but can also be used to perform in-depth performance analysis of the storage cluster. By using advanced load generation tools and performance monitoring software, various actual workloads are simulated to collect detailed performance indicators such as response time, throughput, and error rate. These data will be used to build a performance model for each storage cluster and predict performance under different load conditions, so as to more accurately determine the maximum limit of concurrent operations.

[0050] Different load conditions refer to the varying degrees of read and write requests and data transmission pressure that the storage cluster endures during operation, including low load, medium load, and high load.

[0051] Stress testing is a performance testing method that simulates the operation of a storage cluster under different load conditions to evaluate its performance and stability.

[0052] Step S202 : acquiring real-time monitoring data of the storage cluster, and using a performance model to evaluate and analyze the real-time monitoring data to obtain a current pressure state value of the storage cluster.

[0053] In one embodiment, real-time monitoring data of the storage cluster can be obtained through a preset monitoring module. The monitoring module is a software system module used to monitor the operating status and performance indicators of the storage cluster in real time, enabling timely detection and reporting of abnormal conditions in the storage cluster. In addition to basic hardware resource monitoring, the monitoring module can also integrate intelligent analysis capabilities to identify potential performance bottlenecks and failure risks. For example, machine learning algorithms can analyze historical data to predict future load trends, allowing adjustments to be made in advance to avoid system overload. Furthermore, a real-time alert system is essential, capable of immediately notifying administrators when abnormal conditions are detected and enabling appropriate measures to be taken.

[0054] Real-time monitoring data refers to the operating status and performance metrics of the storage cluster, acquired in real time. This data can be used to assess the current operational status of the storage cluster. This data can be obtained in real time through the monitoring module.

[0055] Evaluation and analysis refers to the processing and analysis of real-time monitoring data based on performance models to evaluate the current stress status and performance of the storage cluster.

[0056] The pressure status value refers to a quantitative indicator obtained through evaluation and analysis that reflects the current pressure status and performance of the storage cluster.

[0057] Step S203: If the pressure state value exceeds a preset pressure threshold, a load variation pattern is extracted from historical data of the storage cluster to generate a load prediction model.

[0058] The pressure threshold refers to a preset critical value of the storage cluster pressure state. When the pressure state value exceeds this threshold, it indicates that the storage cluster may face a performance bottleneck or abnormal condition.

[0059] In one embodiment, a monitoring module may be used to extract load variation patterns from historical data of the storage cluster to generate a load prediction model.

[0060] Historical data refers to the operating status and performance indicators of the storage cluster over the past period of time. It can be used to analyze the load variation patterns of the storage cluster.

[0061] The load variation pattern refers to the pattern and trend of load variation over time during the operation of the storage cluster.

[0062] The load prediction model is a model built based on historical data and load change patterns, used to predict future load changes in the storage cluster.

[0063] Step S204: obtaining the real-time performance index of the storage cluster, and calculating the target concurrency of migrations of the storage cluster based on the real-time performance index and using a load prediction model.

[0064] Among them, real-time performance indicators refer to indicators that reflect the current performance status of the storage cluster, such as response time, throughput, resource utilization, etc.

[0065] The target migration concurrency refers to the optimal migration concurrency that the storage cluster should achieve during data migration, calculated based on the load prediction model.

[0066] Step S205 : Based on the target migration concurrency, adjust the current data migration concurrency and bandwidth limit parameters of the storage cluster to obtain the target data migration concurrency corresponding to the data migration concurrency and the target bandwidth limit parameters corresponding to the bandwidth display parameters.

[0067] The control module is a software system module used to adjust the number of concurrent data migrations and bandwidth limit parameters of the storage cluster.

[0068] In one embodiment, a preset control module can be used to adjust the current data migration concurrency and bandwidth limit parameters of the storage cluster to obtain a target data migration concurrency corresponding to the data migration concurrency and a target bandwidth limit parameter corresponding to the bandwidth display parameter. The control module can not only adjust the data migration concurrency according to the current load situation, but also has predictive capabilities. For example, the migration strategy can be adjusted in advance based on historical data and load prediction models to cope with the upcoming peak load. In addition, the control module can also have a fault-tolerant mechanism that can automatically reschedule when some data migration tasks fail, ensuring the smooth progress of the overall data migration plan.

[0069] The number of concurrent data migration tasks refers to the number of migration tasks performed simultaneously during the data migration process.

[0070] The bandwidth limit parameter refers to a parameter that limits the data transmission bandwidth during the data migration process.

[0071] The target concurrent number of data migrations refers to the actual concurrent number of data migrations achieved by the storage cluster after adjustment by the control module.

[0072] The target bandwidth limit parameter refers to the bandwidth limit parameter actually applied by the storage cluster after adjustment by the control module.

[0073] Step S206 : Using a preset data transmission protocol, migrate the data of the storage cluster according to the target data migration concurrency number and the target bandwidth limit parameters.

[0074] The execution module is a software system module responsible for executing data migration tasks and migrating data from the source storage cluster to the target storage cluster.

[0075] In one embodiment, a preset execution module and a preset data transfer protocol can be used to migrate data in a storage cluster according to the target data migration concurrency and target bandwidth limit parameters. The execution module is highly reliable and flexible, supporting multiple data transfer protocols to adapt to different network environments and storage systems. During the data migration process, strict data integrity checks can be implemented to ensure data accuracy. Furthermore, to improve efficiency, the execution module can also support parallel processing and breakpoint resume functions, ensuring that migration tasks can be quickly resumed in the event of network interruptions or other abnormalities.

[0076] Among them, the data transmission protocol refers to the protocol used to ensure the reliability and integrity of data transmission during the data migration process.

[0077] The data in the storage cluster refers to various types of data stored in the storage cluster, such as user data and system data.

[0078] Data migration refers to the process of migrating data in a storage cluster from one location or system to another.

[0079] The embodiment of the present application can propose an innovative solution to the problems of fixed migration parameters, lagging monitoring mechanism and low migration efficiency in the existing disaster recovery migration process. By obtaining the stress test data of the storage cluster to build a performance model and monitoring the cluster status in real time, the current pressure state can be accurately assessed. When the pressure exceeds the preset threshold, the load change pattern can be extracted from the historical data in a timely manner to generate a load prediction model. The load prediction model is combined with real-time performance indicators to dynamically calculate the target migration concurrency, and then adjust the data migration concurrency and bandwidth limit parameters to achieve intelligent dynamic adjustment of migration parameters. This solution can not only effectively respond to sudden load changes, avoid the continuous deterioration of the underlying cluster performance, reduce the occurrence of slow request timeouts and system anomalies, ensure normal user read and write operations, and improve user experience, but also through real-time monitoring and adjustment mechanisms, it can effectively identify problems before they occur, reducing system risks and uncertainties. In addition, when facing large-scale data migration, this solution can significantly improve migration efficiency, reduce storage pressure and load capacity constraints, and ensure that the disaster recovery migration process is smoother and more efficient.

[0080] In some optional implementations of this embodiment, in step 201, obtaining stress test data of the storage cluster specifically includes the following steps:

[0081] Obtain stress test conditions and test parameters for the storage cluster under different load conditions; perform stress tests on the storage cluster based on the stress test conditions and test parameters under different load conditions to generate performance data for the storage cluster under different load conditions; use a preset machine learning algorithm to deeply mine the performance data to obtain stress test data for the storage cluster.

[0082] The stress test conditions refer to the operating environment and load conditions of the storage cluster set during the stress test.

[0083] Among them, test parameters refer to the parameters used to control the test process and evaluate the test results when performing stress testing.

[0084] Among them, performance data refers to data obtained through stress testing that reflects the performance of the storage cluster.

[0085] Machine learning algorithms are computational methods that learn from training data and automatically improve algorithm performance. They can be used to conduct in-depth mining and analysis of performance data.

[0086] Among them, deep mining refers to the use of machine learning algorithms to conduct in-depth analysis and mining of performance data to discover the potential patterns and trends in storage cluster performance and obtain stress testing data.

[0087] In one example, this embodiment is described in detail below with reference to actual cases in the financial field. The transaction system of a financial institution uses a distributed storage cluster to store transaction data. In order to evaluate the performance of the storage cluster in a high-concurrency transaction scenario, first, different load conditions can be simulated based on historical transaction data (such as 1,000, 2,000, 3,000 transactions per second, etc.). Then, based on these load conditions, corresponding stress test conditions and test parameters are set. A preset performance evaluation and analysis module can be used to stress test the storage cluster and generate performance data under different load conditions. Finally, machine learning algorithms such as support vector machines are used to deeply mine the performance data to obtain stress test data of the storage cluster. The stress test data may include key indicators such as maximum throughput, response time, and resource utilization.

[0088] The embodiments of the present application can achieve a comprehensive evaluation of the performance of the storage cluster by accurately obtaining the storage cluster stress test conditions and test parameters under different load conditions. Under different load conditions, stress testing can generate detailed performance data of the storage cluster under different loads. This data covers multiple key performance indicators such as the storage cluster's throughput, response time, and resource utilization. Furthermore, by using a preset machine learning algorithm to conduct in-depth mining of performance data, potential problems and bottlenecks of the storage cluster during stress testing are revealed, thereby obtaining more accurate stress test data.

[0089] In some optional implementations of this embodiment, step S202, obtaining real-time monitoring data of the storage cluster, specifically includes the following steps:

[0090] Obtain the monitoring rules and monitoring thresholds of the storage cluster; based on the monitoring rules, monitor the storage cluster in real time to obtain various monitoring indicator data of the storage cluster; compare the monitoring indicator data with the monitoring threshold to determine whether there is any abnormality in the storage cluster; if the monitoring indicator data exceeds the monitoring threshold, the monitoring indicator data is used as the real-time monitoring data of the storage cluster.

[0091] Monitoring rules are used to monitor the operating status and performance indicators of the storage cluster, including the selection of monitoring indicators and monitoring frequency.

[0092] The monitoring threshold refers to the preset critical value of the storage cluster operation status and performance indicators. When the monitoring indicator data exceeds this threshold, an alarm can be triggered.

[0093] The monitoring indicator data refers to the real-time data obtained through the monitoring module that reflects the operating status and performance indicators of the storage cluster.

[0094] In one example, this embodiment is described in detail below with reference to actual cases in the financial field. The transaction system of a financial institution uses a high-performance storage cluster to store important data such as transaction records and customer information. In order to ensure the stable operation of the storage cluster, detailed monitoring rules can be formulated first according to the characteristics of the storage cluster and business needs, including monitoring indicators (such as disk usage, memory occupancy, I / O read and write rates, etc.) and monitoring cycles (such as every minute, every hour, etc.). At the same time, based on historical data and experience, reasonable monitoring thresholds are set for each monitoring indicator. Then, a preset monitoring module can be used to monitor the storage cluster in real time. The monitoring module collects the various monitoring indicator data of the storage cluster periodically or in real time according to the monitoring rules, and compares these data with the monitoring thresholds. If a certain monitoring indicator data exceeds its corresponding monitoring threshold, it is considered that there is an abnormality in the storage cluster, and the monitoring indicator data is used as the real-time monitoring data of the storage cluster.

[0095] The embodiments of the present application enable real-time monitoring of the storage cluster status by precisely setting the storage cluster's monitoring rules and monitoring thresholds. Key monitoring indicator data for the storage cluster, such as disk usage and memory occupancy, is efficiently collected, ensuring the comprehensiveness and accuracy of the data. By intelligently comparing monitoring indicator data with monitoring thresholds, abnormal storage cluster conditions can be quickly identified. Once a monitoring indicator exceeds the threshold, the relevant data is immediately marked as real-time monitoring data.

[0096] In some optional implementations of this embodiment, step S203 uses a monitoring module to extract load variation patterns from historical data of the storage cluster and generate a load prediction model, which specifically includes the following steps:

[0097] Obtain historical data of the storage cluster, sample the historical data according to the preset time granularity, and obtain a load change data series; perform feature extraction on the load change data series to obtain key features that reflect the periodicity and trend of the load change data series; and establish a load forecasting model based on the key features using a time series analysis algorithm.

[0098] The time granularity refers to the set time interval or unit time length during the sampling process.

[0099] Sampling refers to the process of extracting load change data from historical data based on a preset time granularity.

[0100] The load change data sequence refers to a set of load change data arranged in chronological order.

[0101] Among them, feature extraction refers to the process of extracting key features that reflect the periodicity and trend of load change data series.

[0102] Among them, key features refer to features that can reflect the main laws and trends of load change data series.

[0103] Among them, the time series analysis algorithm is used to analyze and predict time series data (such as load change data).

[0104] In one example, this embodiment is described in detail below with reference to a practical case in the financial sector. A financial institution's trading system uses a high-performance storage cluster. To accurately predict the storage cluster's load, historical data can first be obtained. This historical data includes information about the storage cluster's load over the past period, such as the number of read and write requests per second and I / O throughput. The historical data is then sampled according to a preset time granularity (e.g., hourly, minutely, etc.) to obtain a load variation data sequence. Next, a monitoring module can be used to extract features from the load variation data sequence. The monitoring module calculates the autocorrelation function and partial autocorrelation function of the load variation data sequence to identify periodic characteristics of load variation, such as daily peak trading hours. Simultaneously, by calculating the trend term coefficient, trend characteristics of load variation are obtained, such as the gradual increase in the storage cluster's load as business volume increases. Finally, based on the extracted key features, a time series analysis algorithm is used to establish a load prediction model. The load prediction model can predict load variations over a period of time in the future based on current load characteristics.

[0105] The embodiment of the present application can achieve accurate prediction of load changes by deeply mining the historical data of the storage cluster. First, the historical data is carefully sampled according to the time granularity to construct a load change data series, which provides a solid foundation for subsequent analysis. Then, the load change data series is extracted, which not only accurately captures the periodic laws of load changes, such as daily or weekly peak hours, but also deeply reveals the trend characteristics of load changes, such as long-term growth or downward trends. Finally, based on these key features, an efficient time series analysis algorithm is used to successfully establish a load prediction model. The load prediction model can predict the load situation of the storage cluster in advance and provide strong support for decisions such as system resource scheduling.

[0106] In some optional implementations of this embodiment, step S204, based on the real-time performance indicators and using the load prediction model, calculates the target concurrency number of migrations of the storage cluster, which specifically includes the following steps:

[0107] The real-time performance indicators are input into the load prediction model to obtain the load change trend prediction results of the storage cluster within the target time. Based on the load change trend prediction results, the target concurrency number of migrations of the storage cluster is calculated.

[0108] The load change trend prediction result refers to the trend and result of predicting the future load change of the storage cluster obtained through the time series analysis algorithm.

[0109] In one example, the storage cluster of a large financial institution is used as an example. This storage cluster handles a large amount of transaction data storage and query tasks, placing extremely high demands on performance and stability. First, real-time performance metrics of the storage cluster are obtained, including but not limited to CPU usage, memory utilization, and disk I / O read / write rates. These real-time performance metrics reflect the current operating status and load of the storage cluster. Next, these real-time performance metrics are input into a pre-trained load prediction model. The load prediction model predicts the load trend of the storage cluster within a target timeframe (for example, within the next week). Based on the predicted load trend, the target migration concurrency for the storage cluster is calculated. This target migration concurrency is the maximum number of migration tasks that can be performed simultaneously while ensuring the performance and stability of the storage cluster. This calculation comprehensively considers factors such as the storage cluster's hardware resources, network bandwidth, and load fluctuations. Complex algorithms and optimization strategies are used to determine a reasonable target migration concurrency.

[0110] The embodiment of the present application can achieve an accurate prediction of the load change trend of the storage cluster within the future target time by inputting real-time performance indicators into the load prediction model. This prediction result not only takes into account the historical load data of the storage cluster, but also incorporates real-time performance indicators, thereby significantly improving the accuracy of the prediction. Based on this prediction result, the target migration concurrency of the storage cluster is further calculated. This value is the maximum number of migration tasks that can be performed simultaneously while ensuring the performance and stability of the storage cluster. By reasonably controlling the number of migration concurrency, the system performance degradation and instability caused by too many migration tasks are effectively avoided. At the same time, the solution of this embodiment also has the ability of dynamic adjustment, and can flexibly adjust the number of migration concurrency according to real-time load changes, thereby further optimizing the resource utilization and overall performance of the storage cluster.

[0111] In some optional implementations of this embodiment, in step S206, before migrating the data of the storage cluster using a preset data transmission protocol and according to the target data migration concurrency number and target bandwidth limit parameters, the following steps are specifically further included:

[0112] Build a visual interface; visualize the real-time progress and status information of data migration to obtain a data migration visual interface.

[0113] In one embodiment, a visual interface is constructed through a configuration module. The configuration module provides an intuitive and easy-to-use user interface (i.e., a data migration visualization interface), allowing users to set complex migration parameters through simple operations. Furthermore, real-time migration status monitoring and detailed logging are essential, allowing users to review migration progress at any time and analyze any issues encountered during the migration process. To improve system maintainability, the configuration module also supports remote management and automated scripts to simplify daily operations.

[0114] The data migration visualization interface is a user interface used to display the migration progress and status information in real time during the data migration process.

[0115] The real-time progress refers to the ratio of the currently completed migration tasks to the total migration tasks or the progress bar during the data migration process.

[0116] Status information refers to information reflecting the status and results of the migration task during the data migration process, such as success, failure, or pause.

[0117] In one example, a storage cluster at a large cloud computing service provider is used. This storage cluster comprises multiple storage nodes, and data needs to be regularly migrated from one node to another to optimize storage performance and resource allocation. First, a pre-defined configuration module is used to develop a data migration visualization interface. This configuration module contains the various parameters and configuration information required for the data migration process. This module allows for convenient setup and adjustment of data migration tasks. Next, the data migration visualization interface is integrated into the storage cluster's management system. This interface uses a graphical design to intuitively display real-time data migration progress and status information. Specifically, the data migration visualization interface displays key information such as the data migration task's start time, estimated completion time, current progress percentage, amount of data migrated, and remaining data. Furthermore, the data migration visualization interface provides a variety of interactive features, such as pausing, accelerating, and reconfiguring the migration process. In practice, the data migration visualization interface can be used to monitor the progress of data migration within the storage cluster in real time.

[0118] The embodiment of the present application can innovatively develop a data migration visualization interface, which plays an important role in the data migration process of the storage cluster. Through the data migration visualization interface, the real-time progress of the data migration can be intuitively seen, including key information such as the amount of data migrated, the migration speed, and the estimated completion time, which greatly improves the transparency and controllability of the data migration process. At the same time, the data migration visualization interface can also display the status information of the data migration in real time, such as whether the migration task is running normally, whether errors or anomalies are encountered, etc., so that potential problems can be discovered and handled in a timely manner to avoid the risk of data loss or migration failure.

[0119] In some optional implementations of this embodiment, in step S206, after migrating the data of the storage cluster using a preset data transmission protocol according to the target data migration concurrency number and the target bandwidth limit parameter, the following steps are further included:

[0120] Obtain breakpoint location information during the data migration process; determine whether the breakpoint location information is empty; if the breakpoint location information is not empty, use a preset verification algorithm to verify the integrity of the migrated data and obtain a verification result; if the verification result shows that the migrated data is incomplete, obtain the corresponding target breakpoint location information from the breakpoint location information; and re-initiate data migration from the breakpoint indicated by the target breakpoint location information.

[0121] The breakpoint location information refers to the location information of data that cannot be successfully migrated due to a failure or interruption during the data migration process.

[0122] The verification algorithm is used to verify the integrity of the migrated data and ensure the reliability of the migration by comparing the consistency of the source data and the target data.

[0123] The verification result refers to the result obtained through the verification algorithm, which reflects the integrity of the migrated data.

[0124] The target breakpoint location information refers to location information of data that needs to be re-migrated, which is determined based on the breakpoint location information during the data migration process.

[0125] Re-initiating data migration at a breakpoint refers to restarting data migration from the breakpoint when a breakpoint location information is encountered during the data migration process.

[0126] In one example, a storage cluster at a large financial institution is used. This storage cluster comprises multiple storage nodes, and data needs to be regularly migrated from one node to another for backup. During the data migration process, the monitoring module first obtains the migration task's breakpoint location information in real time. This breakpoint location information represents the demarcation point between migrated and unmigrated data when the migration task is paused due to various reasons (such as network interruption or storage node failure). If the monitoring module detects that the breakpoint location information is not empty, indicating that there are unfinished migration tasks, the storage cluster's data migration is determined to be incomplete. Next, a verification algorithm is used to verify the integrity of the migrated data. This verification algorithm can be a hash value comparison algorithm or other algorithm that ensures data integrity. The verification result is obtained by calculating the hash value of the migrated data and comparing it with the hash value of the original data. If the verification result indicates that the migrated data is incomplete (i.e., the hash values ​​do not match), this indicates that data loss or corruption occurred during the migration process. After confirming that the migrated data is incomplete, the corresponding target breakpoint location information is obtained from the breakpoint location information. The target breakpoint location information indicates the starting point for re-initiating migration. This is the starting point of the data that was not migrated when the migration task was last paused. By parsing the breakpoint location information, the target breakpoint location information can be determined, allowing migration to be re-initiated from that location. Finally, the execution module is used to re-initiate data migration from the breakpoint indicated by the target breakpoint location information. The execution module is the component responsible for executing the migration task. Based on the target breakpoint location information, it re-migrates data from the breakpoint until the migration task is completed. During the re-migration process, the monitoring module can continue to obtain breakpoint location information in real time to ensure the smooth progress of the migration task.

[0127] The embodiment of the present application can achieve accurate monitoring of the progress of data migration by obtaining the breakpoint location information during the data migration process. When it is detected that the breakpoint location information is not empty, it can be immediately determined that the data migration of the storage cluster is not completed, so that timely measures can be taken to deal with it. At the same time, a verification algorithm is used to verify the integrity of the migration data to ensure the accuracy and reliability of the migration data. If the verification result shows that the migration data is incomplete, the present technical solution can obtain the corresponding target breakpoint location information from the breakpoint location information, and re-initiate data migration from the breakpoint through the execution module, thereby avoiding migration failure due to data loss or damage.

[0128] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned stress testing data and real-time monitoring data, the source cluster information of the above-mentioned source cluster, the target cluster information of the target cluster, and the migration strategy can also be stored in a blockchain node.

[0129] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0130] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0131] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0132] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data migration device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0133] like Figure 3 As shown, the data migration device 400 of this embodiment includes: a first acquisition module 401, a second acquisition module 402, an extraction module 403, a calculation module 404, an adjustment module 405 and a data migration module 406.

[0134] A first acquisition module 401 is configured to acquire stress test data of a storage cluster and construct a performance model of the storage cluster based on the stress test data. The stress test data is acquired by performing stress tests on the storage cluster under different load conditions.

[0135] The second acquisition module 402 is used to acquire real-time monitoring data of the storage cluster, and use the performance model to evaluate and analyze the real-time monitoring data to obtain the current pressure status value of the storage cluster;

[0136] Extraction module 403, configured to extract load variation patterns from historical data of the storage cluster and generate a load prediction model if the pressure state value exceeds a preset pressure threshold;

[0137] The calculation module 404 is used to obtain the real-time performance indicators of the storage cluster and calculate the target concurrency of migration of the storage cluster based on the real-time performance indicators and using a load prediction model;

[0138] Adjustment module 405, configured to adjust the current data migration concurrency and bandwidth limit parameters of the storage cluster based on the target migration concurrency, and obtain the target data migration concurrency corresponding to the data migration concurrency and the target bandwidth limit parameters corresponding to the bandwidth display parameters;

[0139] The data migration module 406 is configured to migrate the data of the storage cluster using a preset data transmission protocol according to target data migration concurrency and target bandwidth limit parameters.

[0140] The embodiment of the present application can propose an innovative solution to the problems of fixed migration parameters, lagging monitoring mechanism and low migration efficiency in the existing disaster recovery migration process. By obtaining the stress test data of the storage cluster to build a performance model and monitoring the cluster status in real time, the current pressure state can be accurately assessed. When the pressure exceeds the preset threshold, the load change pattern can be extracted from the historical data in a timely manner to generate a load prediction model. The load prediction model is combined with real-time performance indicators to dynamically calculate the target migration concurrency, and then adjust the data migration concurrency and bandwidth limit parameters to achieve intelligent dynamic adjustment of migration parameters. This solution can not only effectively respond to sudden load changes, avoid the continuous deterioration of the underlying cluster performance, reduce the occurrence of slow request timeouts and system anomalies, ensure normal user read and write operations, and improve user experience, but also through real-time monitoring and adjustment mechanisms, it can effectively identify problems before they occur, reducing system risks and uncertainties. In addition, when facing large-scale data migration, this solution can significantly improve migration efficiency, reduce storage pressure and load capacity constraints, and ensure that the disaster recovery migration process is smoother and more efficient.

[0141] In one embodiment, the first acquisition module 401 includes:

[0142] The first acquisition submodule is used to obtain stress test conditions and test parameters of the storage cluster under different load conditions;

[0143] The test submodule is used to perform stress tests on the storage cluster under different load conditions based on stress test conditions and test parameters, and generate performance data of the storage cluster under different load conditions;

[0144] The mining submodule is used to use a preset machine learning algorithm to deeply mine performance data and obtain stress testing data for the storage cluster.

[0145] The embodiments of the present application can achieve a comprehensive evaluation of the performance of the storage cluster by accurately obtaining the storage cluster stress test conditions and test parameters under different load conditions. Under different load conditions, stress testing can generate detailed performance data of the storage cluster under different loads. This data covers multiple key performance indicators such as the storage cluster's throughput, response time, and resource utilization. Furthermore, by using a preset machine learning algorithm to conduct in-depth mining of performance data, potential problems and bottlenecks of the storage cluster during stress testing are revealed, thereby obtaining more accurate stress test data.

[0146] In one embodiment, the second obtaining module 402 includes:

[0147] The second acquisition submodule is used to obtain the monitoring rules and monitoring thresholds of the storage cluster;

[0148] The monitoring submodule is used to monitor the storage cluster in real time based on monitoring rules and obtain various monitoring indicator data of the storage cluster;

[0149] The comparison submodule is used to compare the monitoring indicator data with the monitoring threshold to determine whether there are any abnormalities in the storage cluster;

[0150] As a submodule, if the monitoring indicator data exceeds the monitoring threshold, the monitoring indicator data will be used as real-time monitoring data of the storage cluster.

[0151] The embodiments of the present application enable real-time monitoring of the storage cluster status by precisely setting the storage cluster's monitoring rules and monitoring thresholds. Key monitoring indicator data for the storage cluster, such as disk usage and memory occupancy, is efficiently collected, ensuring the comprehensiveness and accuracy of the data. By intelligently comparing monitoring indicator data with monitoring thresholds, abnormal storage cluster conditions can be quickly identified. Once a monitoring indicator exceeds the threshold, the relevant data is immediately marked as real-time monitoring data.

[0152] In one embodiment, the extraction module 403 includes:

[0153] The sampling submodule is used to obtain historical data of the storage cluster, sample the historical data according to the preset time granularity, and obtain the load change data sequence;

[0154] The extraction submodule is used to extract features from the load change data sequence and obtain key features that reflect the periodicity and trend of the load change data sequence;

[0155] A submodule is established to build a load forecasting model based on key features and using time series analysis algorithm.

[0156] The embodiment of the present application can achieve accurate prediction of load changes by deeply mining the historical data of the storage cluster. First, the historical data is carefully sampled according to the time granularity to construct a load change data series, which provides a solid foundation for subsequent analysis. Then, the load change data series is extracted, which not only accurately captures the periodic laws of load changes, such as daily or weekly peak hours, but also deeply reveals the trend characteristics of load changes, such as long-term growth or downward trends. Finally, based on these key features, an efficient time series analysis algorithm is used to successfully establish a load prediction model. The load prediction model can predict the load situation of the storage cluster in advance and provide strong support for decisions such as system resource scheduling.

[0157] In one embodiment, the calculation module 404 includes:

[0158] The input submodule is used to input real-time performance indicators into the load prediction model to obtain the load change trend prediction results of the storage cluster within the target time;

[0159] The calculation submodule is used to calculate the target concurrency of migrations in the storage cluster based on the load change trend prediction results.

[0160] The embodiment of the present application can achieve an accurate prediction of the load change trend of the storage cluster within the future target time by inputting real-time performance indicators into the load prediction model. This prediction result not only takes into account the historical load data of the storage cluster, but also incorporates real-time performance indicators, thereby significantly improving the accuracy of the prediction. Based on this prediction result, the target migration concurrency of the storage cluster is further calculated. This value is the maximum number of migration tasks that can be performed simultaneously while ensuring the performance and stability of the storage cluster. By reasonably controlling the number of migration concurrency, the system performance degradation and instability caused by too many migration tasks are effectively avoided. At the same time, the solution of this embodiment also has the ability of dynamic adjustment, and can flexibly adjust the number of migration concurrency according to real-time load changes, thereby further optimizing the resource utilization and overall performance of the storage cluster.

[0161] In one embodiment, the data migration apparatus further includes:

[0162] Building modules for building visualization interfaces;

[0163] The visualization module is used to visualize the real-time progress and status information of data migration to obtain a data migration visualization interface.

[0164] The embodiment of the present application can innovatively develop a data migration visualization interface, which plays an important role in the data migration process of the storage cluster. Through the data migration visualization interface, the real-time progress of the data migration can be intuitively seen, including key information such as the amount of data migrated, the migration speed, and the estimated completion time, which greatly improves the transparency and controllability of the data migration process. At the same time, the data migration visualization interface can also display the status information of the data migration in real time, such as whether the migration task is running normally, whether errors or anomalies are encountered, etc., so that potential problems can be discovered and handled in a timely manner to avoid the risk of data loss or migration failure.

[0165] In one embodiment, the data migration apparatus further includes:

[0166] The third acquisition module is used to obtain breakpoint location information during the data migration process;

[0167] A judgment module is used to judge whether the breakpoint location information is empty;

[0168] A verification module is used to verify the integrity of the migrated data using a preset verification algorithm if the breakpoint location information is not empty, and obtain a verification result;

[0169] An information acquisition module, configured to acquire corresponding target breakpoint location information from the breakpoint location information if the verification result shows that the migration data is incomplete;

[0170] The initiating module is used to re-initiate data migration from the breakpoint indicated by the target breakpoint position information.

[0171] The embodiment of the present application can achieve accurate monitoring of the progress of data migration by obtaining the breakpoint location information during the data migration process. When it is detected that the breakpoint location information is not empty, it can be immediately determined that the data migration of the storage cluster is not completed, so that timely measures can be taken to deal with it. At the same time, a verification algorithm is used to verify the integrity of the migration data to ensure the accuracy and reliability of the migration data. If the verification result shows that the migration data is incomplete, the present technical solution can obtain the corresponding target breakpoint location information from the breakpoint location information, and re-initiate data migration from the breakpoint through the execution module, thereby avoiding migration failure due to data loss or damage.

[0172] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0173] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0174] Computer devices can be desktop computers, laptops, PDAs, cloud servers, etc. Computer devices can interact with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0175] Memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, memory 61 may be an internal storage unit of computer device 6, such as the hard disk or memory of computer device 6. In other embodiments, memory 61 may also be an external storage device of computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash memory card, etc. equipped on computer device 6. Of course, memory 61 may also include both internal storage units and external storage devices of computer device 6. In this embodiment, memory 61 is generally used to store the operating system and various application software installed on computer device 6, such as computer-readable instructions for data migration methods. In addition, memory 61 may also be used to temporarily store various types of data that have been output or are about to be output.

[0176] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for executing the data migration method.

[0177] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0178] The embodiment of the present application can propose an innovative solution to the problems of fixed migration parameters, lagging monitoring mechanism and low migration efficiency in the existing disaster recovery migration process. By obtaining the stress test data of the storage cluster to build a performance model and monitoring the cluster status in real time, the current pressure state can be accurately assessed. When the pressure exceeds the preset threshold, the load change pattern can be extracted from the historical data in a timely manner to generate a load prediction model. The load prediction model is combined with real-time performance indicators to dynamically calculate the target migration concurrency, and then adjust the data migration concurrency and bandwidth limit parameters to achieve intelligent dynamic adjustment of migration parameters. This solution can not only effectively respond to sudden load changes, avoid the continuous deterioration of the underlying cluster performance, reduce the occurrence of slow request timeouts and system anomalies, ensure normal user read and write operations, and improve user experience, but also through real-time monitoring and adjustment mechanisms, it can effectively identify problems before they occur, reducing system risks and uncertainties. In addition, when facing large-scale data migration, this solution can significantly improve migration efficiency, reduce storage pressure and load capacity constraints, and ensure that the disaster recovery migration process is smoother and more efficient.

[0179] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data migration method as described above.

[0180] The embodiment of the present application can propose an innovative solution to the problems of fixed migration parameters, lagging monitoring mechanism and low migration efficiency in the existing disaster recovery migration process. By obtaining the stress test data of the storage cluster to build a performance model and monitoring the cluster status in real time, the current pressure state can be accurately assessed. When the pressure exceeds the preset threshold, the load change pattern can be extracted from the historical data in a timely manner to generate a load prediction model. The load prediction model is combined with real-time performance indicators to dynamically calculate the target migration concurrency, and then adjust the data migration concurrency and bandwidth limit parameters to achieve intelligent dynamic adjustment of migration parameters. This solution can not only effectively respond to sudden load changes, avoid the continuous deterioration of the underlying cluster performance, reduce the occurrence of slow request timeouts and system anomalies, ensure normal user read and write operations, and improve user experience, but also through real-time monitoring and adjustment mechanisms, it can effectively identify problems before they occur, reducing system risks and uncertainties. In addition, when facing large-scale data migration, this solution can significantly improve migration efficiency, reduce storage pressure and load capacity constraints, and ensure that the disaster recovery migration process is smoother and more efficient.

[0181] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0182] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0183] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A data migration method, characterized in that: The steps include: Obtaining stress test data of a storage cluster, and building a performance model of the storage cluster based on the stress test data, wherein the stress test data is obtained by performing stress tests on the storage cluster under different load conditions; Acquire real-time monitoring data of the storage cluster, use the performance model to evaluate and analyze the real-time monitoring data, and obtain a current pressure state value of the storage cluster; If the pressure state value exceeds a preset pressure threshold, extracting a load variation rule from historical data of the storage cluster to generate a load prediction model; Acquire the real-time performance index of the storage cluster, and calculate the target concurrent number of migrations of the storage cluster based on the real-time performance index and using the load prediction model; Based on the target concurrency number of migrations, the current concurrency number of data migrations and the bandwidth limit parameter of the storage cluster are adjusted to obtain the target concurrency number of data migrations corresponding to the concurrency number of data migrations and the target bandwidth limit parameter corresponding to the bandwidth display parameter; The data of the storage cluster is migrated by adopting a preset data transmission protocol according to the target data migration concurrency number and the target bandwidth limit parameter.

2. The method according to claim 1, characterized in that: The step of obtaining the stress test data of the storage cluster specifically includes: Obtain stress test conditions and test parameters for storage clusters under different load conditions; Under different load conditions, based on the stress test conditions and the test parameters, stress testing the storage cluster to generate performance data of the storage cluster under different load conditions; A preset machine learning algorithm is used to deeply mine the performance data to obtain stress testing data of the storage cluster.

3. The method according to claim 1, characterized in that The step of obtaining the real-time monitoring data of the storage cluster specifically includes: Obtaining monitoring rules and monitoring thresholds of the storage cluster; Based on the monitoring rules, the storage cluster is monitored in real time to obtain various monitoring indicator data of the storage cluster; Comparing the monitoring indicator data with the monitoring threshold to determine whether there is an abnormality in the storage cluster; If the monitoring indicator data exceeds the monitoring threshold, the monitoring indicator data is used as real-time monitoring data of the storage cluster.

4. The method according to claim 1, characterized in that The step of extracting load variation rules from the historical data of the storage cluster and generating a load prediction model specifically includes: Acquire historical data of the storage cluster, sample the historical data according to a preset time granularity, and obtain a load change data sequence; Performing feature extraction on the load change data sequence to obtain key features reflecting the periodicity and trend of the load change data sequence; According to the key characteristics, a load forecasting model is established by using a time series analysis algorithm.

5. The method according to claim 1, characterized in that The step of calculating the target concurrent number of migrations of the storage cluster based on the real-time performance indicator and using the load prediction model specifically includes: Inputting the real-time performance indicator into the load prediction model to obtain a load change trend prediction result of the storage cluster within a target time; Based on the load change trend prediction result, a target concurrency number of migrations of the storage cluster is calculated.

6. The method according to claim 1, characterized in that Before the step of adopting a preset data transmission protocol to migrate the data of the storage cluster according to the target data migration concurrency number and the target bandwidth limit parameter, the step further includes: Build a visual interface; The real-time progress and status information of the data migration are visualized to obtain a data migration visualization interface.

7. The method according to claim 1, characterized in that After the step of adopting the preset data transmission protocol and performing data migration on the data of the storage cluster according to the target data migration concurrency number and the target bandwidth limit parameter, the step further includes: Obtaining breakpoint location information during the data migration process; Determine whether the breakpoint location information is empty; If the breakpoint location information is not empty, a preset verification algorithm is used to verify the integrity of the migration data to obtain a verification result; If the verification result shows that the migration data is incomplete, obtaining corresponding target breakpoint position information from the breakpoint position information; Re-initiate data migration from the breakpoint indicated by the target breakpoint location information.

8. A data migration device, characterized in that: include: A first acquisition module is used to acquire stress test data of a storage cluster and build a performance model of the storage cluster based on the stress test data, wherein the stress test data is acquired by performing stress tests on the storage cluster under different load conditions; A second acquisition module is used to acquire real-time monitoring data of the storage cluster, and use the performance model to evaluate and analyze the real-time monitoring data to obtain a current pressure state value of the storage cluster; An extraction module, configured to extract a load variation rule from historical data of the storage cluster and generate a load prediction model if the pressure state value exceeds a preset pressure threshold; A calculation module, used to obtain the real-time performance index of the storage cluster, and calculate the target concurrency number of migrations of the storage cluster based on the real-time performance index and using the load prediction model; An adjustment module, configured to adjust the current concurrent number of data migration and the bandwidth limit parameter of the storage cluster based on the target concurrent number of migration, and obtain the target concurrent number of data migration corresponding to the concurrent number of data migration and the target bandwidth limit parameter corresponding to the bandwidth display parameter; The data migration module is used to adopt a preset data transmission protocol to migrate the data of the storage cluster according to the target data migration concurrency number and the target bandwidth limit parameter.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

Images