Data migration method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data migration method and device, electronic equipment and a storage medium, and the method comprises the steps: collecting sample data, and carrying out the preprocessing of the sample data, and obtaining a preprocessed data set; determining output sample data of an update gate of a preset deep learning model and an output value of a reset gate of the preset deep learning model according to the preset deep learning model and the preprocessed data set, and determining time sequence features according to the output sample data of the update gate and the output value of the reset gate; determining a feature data set according to the time sequence features and a preset time sequence convolutional neural network; training a preset model according to the feature data set to obtain a target prediction model; and acquiring real-time data of the system, and determining a scheduling strategy of data migration of the system according to the real-time data and the target prediction model. Therefore, when data migration prediction is carried out according to the target prediction model obtained through training, the prediction efficiency and reliability are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data migration method, a data migration device, an electronic device and a computer-readable storage medium. Background Art

[0002] In the big data processing and cloud computing environment, data migration and synchronization is a crucial task. With the rapid growth of data volume and the diversification of data sources, traditional data migration scheduling methods face many challenges. The rapid development of deep learning technology provides a new solution for intelligent scheduling in data migration and synchronization. Deep learning models are used to predict system load and data change trends in data migration and synchronization, and to achieve intelligent scheduling of data migration and synchronization tasks. However, the deep learning models in related technologies are not efficient and reliable when migrating and synchronizing data. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a data migration method, a data migration device, an electronic device and a computer-readable storage medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problem, an embodiment of the present invention discloses a data migration method, which includes:

[0005] Collect sample data, and preprocess the sample data to obtain a preprocessed data set;

[0006] Determine, according to a preset deep learning model and the preprocessed data set, output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model, and determine a timing feature according to the output sample data of the update gate and the output value of the reset gate;

[0007] Determine a feature data set according to the time series features and a preset time series convolutional neural network;

[0008] Training a preset model according to the feature data set to obtain a target prediction model;

[0009] Real-time data of the system is acquired, and a scheduling strategy for data migration of the system is determined based on the real-time data and the target prediction model.

[0010] Optionally, the determining, according to the preset deep learning model and the preprocessed data set, output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model, and determining the timing features according to the output sample data of the update gate and the timing features includes:

[0011] Determine the data sample of the current time step and the hidden state of the previous time step from the preprocessed data set by using a preset deep learning model;

[0012] Determine output sample data of the update gate according to a first weight matrix of the update gate of the preset deep learning model, a bias term of the update gate, a data sample of the current time step, and a hidden state of the previous time step;

[0013] Determining an output value of a reset gate of the preset deep learning model according to a second weight matrix of the update gate, a bias term of the update gate, a data sample of the current time step, and a hidden state of the previous time step;

[0014] Determine a candidate hidden state for the current time step according to an output value of a reset gate of the preset deep learning model, a data sample of the current time step, and a hidden state of the previous time step;

[0015] Determine the hidden state of the next time step according to the candidate hidden state of the current time step, the output sample data of the update gate, and the hidden state of the previous time step;

[0016] Determine the timing characteristics according to the hidden state of the next time step and the preset deep learning model.

[0017] Optionally, the training of a preset model according to the feature data set to obtain a target prediction model includes:

[0018] Aggregating the feature data set into a four-dimensional stereo feature according to a preset dimensional aggregation function, a preset training weight, and a preset training bias;

[0019] According to the four-dimensional stereo features, the preset model is trained to obtain a target prediction model.

[0020] Optionally, the training of a preset model according to the four-dimensional stereo features to obtain a target prediction model includes:

[0021] Determining sample transposed data information, positive sample information, and negative sample information according to the four-dimensional stereo features;

[0022] The preset model is trained according to the sample transposed data information, the positive sample information and the negative sample information to obtain a target prediction model.

[0023] Optionally, preprocessing the sample data to obtain a preprocessed data set includes:

[0024] determining missing values ​​in the sample data;

[0025] The missing values ​​are filled using the mean filling method to obtain a cleaned data set;

[0026] Performing data enhancement processing on the cleaned data set to obtain an enhanced data set;

[0027] Performing data conversion processing on the enhanced data set to obtain a converted data set;

[0028] Data aggregation processing is performed on the converted data set to obtain a preprocessed data set.

[0029] Optionally, performing data conversion processing on the enhanced data set to obtain a converted data set includes:

[0030] Determine the maximum value and the minimum value of the data in the enhanced data set;

[0031] Obtaining a normalized data set according to the enhanced data set, the data maximum value, and the data minimum value;

[0032] Determine the time series data in the enhanced data set, and process the time series data using a time sliding window method to generate input and output sample data;

[0033] A converted data set is obtained according to the normalized data set and the input and output sample data.

[0034] Optionally, performing data enhancement processing on the cleaned data set to obtain an enhanced data set includes:

[0035] Set the strategy search space;

[0036] Determining a target enhancement strategy from the strategy search space by using a preset search algorithm;

[0037] The target enhancement strategy is used to perform data enhancement processing on the cleaned data set to obtain an enhanced data set.

[0038] The embodiment of the present invention further discloses a data migration device, the device comprising:

[0039] A preprocessing module is used to collect sample data and preprocess the sample data to obtain a preprocessed data set;

[0040] A first determination module is used to determine, according to a preset deep learning model and the preprocessed data set, output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model, and determine a timing feature according to the output sample data of the update gate and the output value of the reset gate;

[0041] A second determination module is used to determine a feature data set according to the time series features and a preset time series convolutional neural network;

[0042] A training module, used to train a preset model according to the feature data set to obtain a target prediction model;

[0043] The third determination module is used to obtain real-time data of the system and determine a scheduling strategy for data migration of the system based on the real-time data and the target prediction model.

[0044] Optionally, the first determining module includes:

[0045] A first determination submodule is used to determine the data sample of the current time step and the hidden state of the previous time step from the preprocessed data set by using a preset deep learning model;

[0046] A second determination submodule, used to determine the output sample data of the update gate according to the first weight matrix of the update gate of the preset deep learning model, the bias term of the update gate, the data sample of the current time step, and the hidden state of the previous time step;

[0047] A third determination submodule, configured to determine an output value of a reset gate of the preset deep learning model according to a second weight matrix of the update gate, a bias term of the update gate, a data sample of the current time step, and a hidden state of the previous time step;

[0048] A fourth determination submodule, configured to determine a candidate hidden state for the current time step according to an output value of a reset gate of the preset deep learning model, a data sample of the current time step, and a hidden state of the previous time step;

[0049] a fifth determination submodule, configured to determine a hidden state for a next time step based on the candidate hidden state for the current time step, the output sample data of the update gate, and the hidden state for the previous time step;

[0050] The sixth determination submodule is used to determine the timing characteristics according to the hidden state of the next time step and the preset deep learning model.

[0051] Optionally, the training module includes:

[0052] A first aggregation submodule, configured to aggregate the feature data set into a four-dimensional stereo feature according to a preset dimensional aggregation function, a preset training weight, and a preset training bias;

[0053] The training submodule is used to train the preset model according to the four-dimensional stereo features to obtain a target prediction model.

[0054] Optionally, the training submodule includes:

[0055] A first determining unit, used to determine sample transposed data information, positive sample information and negative sample information according to the four-dimensional stereo feature;

[0056] The training unit is used to train a preset model according to the sample transposed data information, the positive sample information and the negative sample information to obtain a target prediction model.

[0057] Optionally, the preprocessing module includes:

[0058] A seventh determination submodule, used to determine missing values ​​in the sample data;

[0059] A filling submodule is used to fill the missing values ​​by using a mean filling method to obtain a cleaned data set;

[0060] An enhancement submodule, used to perform data enhancement processing on the cleaned data set to obtain an enhanced data set;

[0061] A conversion submodule, used to perform data conversion processing on the enhanced data set to obtain a converted data set;

[0062] The second aggregation submodule is used to perform data aggregation processing on the converted data set to obtain a preprocessed data set.

[0063] Optionally, the conversion submodule includes:

[0064] A second determining unit, used to determine a maximum value and a minimum value of data in the enhanced data set;

[0065] A normalization unit, used to obtain a normalized data set according to the enhanced data set, the data maximum value and the data minimum value;

[0066] A generating unit, used to determine the time series data in the enhanced data set, and process the time series data in a time sliding window manner to generate input and output sample data;

[0067] The obtaining unit is used to obtain a converted data set according to the normalized data set and the input and output sample data.

[0068] Optionally, the enhancer module comprises:

[0069] A setting unit, used for setting a strategy search space;

[0070] A third determining unit, configured to determine a target enhancement strategy from the strategy search space by using a preset search algorithm;

[0071] The enhancement unit is used to perform data enhancement processing on the cleaned data set through the target enhancement strategy to obtain an enhanced data set.

[0072] The present invention also discloses a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data migration method when executing the computer program.

[0073] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data migration method described above are implemented.

[0074] The embodiments of the present invention include the following advantages:

[0075] In the embodiment of the present invention, sample data is first collected and preprocessed to obtain a preprocessed data set. Then, according to the preset deep learning model and the preprocessed data set, the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model are determined. According to the output sample data of the update gate and the output value of the reset gate, the timing feature is determined. The preset deep learning model of the present invention only includes the update gate and the reset gate. Compared with the learning model in the related art, the structure is simpler and the calculation efficiency is higher. Then, according to the timing feature and the preset timing convolutional neural network, the feature data set is determined, and the preset model is trained according to the feature data set to obtain the target prediction model. Finally, the real-time data of the system is obtained, and according to the real-time data and the target prediction model, the scheduling strategy of the data migration of the system is determined. After the sample data is preprocessed, the preset deep learning model and the preset timing convolutional neural network are used to determine the data set for training the preset model, so that when the target prediction model obtained by training is used for data migration prediction, the efficiency and reliability of its prediction are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is a flowchart of a data migration method provided by an embodiment of the present invention;

[0077] Figure 2 It is a schematic diagram of a data migration framework provided by an embodiment of the present invention;

[0078] Figure 3 It is a structural block diagram of a data migration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0079] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0080] In the related art, when the current deep learning model is used to predict data migration and realize intelligent scheduling of data migration and synchronization tasks, its efficiency and reliability are poor. In order to solve the above technical problems, the present invention provides a data migration method, the core concept of which is to first collect sample data and preprocess the sample data to obtain a preprocessed data set, and then determine the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model according to the preset deep learning model and the preprocessed data set, and determine the timing characteristics according to the output sample data of the update gate and the output value of the reset gate. The preset deep learning model of the present invention only includes the update gate and the reset gate. Compared with the learning model in the related art, the structure is simpler and the calculation efficiency is higher. Then, based on the time series characteristics and the preset time series convolutional neural network, the feature data set is determined, and then the preset model is trained based on the feature data set to obtain the target prediction model. Finally, the real-time data of the system is obtained, and the scheduling strategy for data migration of the system is determined based on the real-time data and the target prediction model. After preprocessing the sample data, the data set for training the preset model is determined through the preset deep learning model and the preset time series convolutional neural network, so that the target prediction model obtained by training can improve its prediction efficiency and reliability when performing data migration prediction.

[0081] Reference Figure 1 , shows a flow chart of a data migration method provided by an embodiment of the present invention, and the method may specifically include the following steps:

[0082] Step 101: collect sample data and preprocess the sample data to obtain a preprocessed data set.

[0083] In an embodiment of the present invention, a target prediction model can be used to determine the scheduling strategy of data migration of the system, and the target prediction model can be obtained by training a preset model based on sample data. Before training the preset model, sample data can be collected first, and the sample data in the present invention can include: system indicator data, task data and historical records.

[0084] In the present invention, the system indicator data may include CPU usage, memory usage, disk input and output data, network bandwidth, etc. CPU usage can reflect the usage of the system processor and is an important indicator for evaluating the system load. Memory usage can reflect the usage of the system memory and affect the operating efficiency of the system. Disk input and output data includes read and write rates and input and output operation times, reflecting the usage of the disk. Network bandwidth may include upload bandwidth and download bandwidth, reflecting the usage of network resources.

[0085] In the present invention, when collecting system indicator data, Prometheus can be used to collect system indicator data. Prometheus is an open source system monitoring and alarm tool suitable for collecting and storing system indicator data. It obtains system indicator data from the target endpoint (Node Exporter) in a pull mode and stores it in a time series database. NodeExporter is a component in the Prometheus ecosystem, which is installed on each node that needs to be monitored to collect system indicators such as CPU usage, memory usage, disk input and output data, and network bandwidth.

[0086] In the present invention, task data may include data volume, data type, data change frequency, etc. Data volume may represent the data size during data migration or synchronization, directly affecting the execution time of data migration, and data type may include structured data, semi-structured data, and unstructured data. Data change frequency may represent the update frequency of data, affecting the size of incremental data and migration frequency.

[0087] In the present invention, when collecting task data, a preset database and an API (Application Programming Interface) interface can be used to collect task data. Specifically, a database connector can be used to extract task data from a preset relational database, and an API interface can be called to extract task data from a preset NoSQL database and other data sources.

[0088] In the present invention, historical record data may include task execution time and task success rate, etc. The task execution time may indicate the start time and end time of each task, the task success rate may indicate the statistical information of task success and failure, and the failure reason may indicate the specific reason for the task failure, such as network failure, system overload, etc.

[0089] In the present invention, when collecting historical records, a unified scheduling platform can be used to obtain historical records, and a preset log management system can be used to collect and analyze task execution logs to obtain detailed execution records of tasks.

[0090] In one embodiment, preprocessing the sample data to obtain a preprocessed data set may include: determining missing values ​​in the sample data; filling the missing values ​​with mean filling to obtain a cleaned data set; performing data enhancement processing on the cleaned data set to obtain an enhanced data set; performing data conversion processing on the enhanced data set to obtain a converted data set; performing data aggregation processing on the converted data set to obtain a preprocessed data set.

[0091] Specifically, the preprocessing in the present invention may include data cleaning, data enhancement processing, data conversion processing and data aggregation processing. In the present invention, since there are many types of data in the sample data, there may be missing values ​​in the data. The missing values ​​in the sample data can be determined first, and then the missing values ​​can be filled by using the mean filling method to obtain a cleaned data set to ensure the integrity of the data.

[0092] In one embodiment, performing data enhancement processing on the cleaned data set to obtain an enhanced data set may include: setting a policy search space; determining a target enhancement policy from the policy search space through a preset search algorithm; and performing data enhancement processing on the cleaned data set through the target enhancement policy to obtain an enhanced data set.

[0093] The present invention can use AutoAugment to automatically search for target enhancement strategies. When performing data enhancement processing, a strategy search space can be set first. The strategy search space can include multiple enhancement strategies, each enhancement strategy can be composed of multiple sub-strategies, and a sub-strategy is randomly selected for each three-dimensional view in each mini-batch. Each sub-strategy can include two operators, and each operator is a two-dimensional data processing function, such as: translation, rotation, shearing, and the probability and size corresponding to the function. Then, a target enhancement strategy is determined from the strategy search space through a preset search algorithm. The target enhancement strategy can be the optimal enhancement strategy. Through the target enhancement strategy, the cleaned data is enhanced to improve the model accuracy.

[0094] In one embodiment, data conversion processing is performed on the enhanced data set to obtain a converted data set, which may include: determining the maximum data value and the minimum data value in the enhanced data set; obtaining a normalized data set based on the enhanced data set, the maximum data value and the minimum data value; determining the time series data in the enhanced data set, and processing the time series data using a time sliding window method to generate input and output sample data; and obtaining a converted data set based on the normalized data set and the input and output sample data.

[0095] The present invention can convert the data of two different sets in the cleaned data set into the same scale to meet the input requirements of the model, which is helpful to improve the performance and stability of the model. The enhanced data set of the present invention can include positive samples and negative samples. For the positive samples, the minimum-maximum normalization method can be used to first determine the maximum value and the minimum value of the data in the enhanced data set, and then obtain the normalized data set according to the enhanced data set, the maximum value and the minimum value of the data. The normalized data set can be obtained by the following formula (1):

[0096] X1=(x-min) / (max-min) Formula (1)

[0097] Among them, X1 represents the normalized data set, x represents the enhanced data set, max represents the maximum value of the data, and min represents the minimum value of the data.

[0098] In the present invention, the enhanced data set contains a large amount of time series data. The time series data can be processed by a time sliding window method to generate input and output sample data, and then the obtained normalized data set and input and output sample data are used as the converted data set. For negative samples, sliding windows of different sizes and steps can be used, and Z- sco r e The standardization method is used to process the data to obtain the input and output sample data. Specifically, the input and output sample data can be obtained by the following formula (2):

[0099]

[0100] Among them, X2 represents the input and output sample data, x represents the enhanced data set, μ represents the mean of the enhanced data set, and σ represents the sample standard deviation of the enhanced data set.

[0101] In an embodiment of the present invention, after obtaining the converted data set, data aggregation processing can be performed on the converted data set to obtain a preprocessed data set. During the data aggregation processing, the data can be aggregated according to different time granularities, such as seconds, minutes, and hours, according to actual needs. This can reduce the amount of data and improve the efficiency of model training.

[0102] Step 102: determine the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model according to the preset deep learning model and the preprocessed data set, and determine the timing characteristics according to the output sample data of the update gate and the output value of the reset gate.

[0103] In an embodiment of the present invention, after obtaining the preprocessed data set, feature extraction processing can be performed to select the owner data that is most useful to the model, reduce the dimension of the data, and improve the training efficiency and prediction performance of the model. The present invention can determine the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model based on the preset deep learning model and the preprocessed data set, and determine the timing characteristics according to the output sample data of the update gate and the output value of the reset gate.

[0104] In the present invention, the preset deep learning model can be a GRU (Gated Recurrent Unit) model, which is a recurrent neural network (RNN) variant for sequence data processing, and is intended to solve the gradient vanishing and gradient exploding problems of traditional RNN in long sequence data processing. GRU introduces two gating mechanisms: update gate and reset gate. These gating mechanisms enable GRU to better capture long-term dependencies while simplifying the complexity of the model. Update gate: determines how much information of the current time step needs to be passed to the next time step. This is equivalent to the memory of past information. Reset gate: controls how much information passed from the previous time step should be forgotten or reset. This is equivalent to selective forgetting of current information. Compared with LSTM (Long Short-Term Memory Network), the GRU model has a simpler structure and higher computational efficiency because it has only two gates, while LSTM has three gates (input gate, forget gate and output gate). Despite the simplified structure, the GRU model performs comparable to LSTM on many tasks, and sometimes even better.

[0105] In one embodiment, according to a preset deep learning model and a preprocessed data set, output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model are determined, and according to the output sample data of the update gate and the timing characteristics are determined, including: determining the data sample of the current time step and the hidden state of the previous time step from the preprocessed data set through the preset deep learning model; determining the output sample data of the update gate according to the first weight matrix of the update gate of the preset deep learning model, the bias item of the update gate, the data sample of the current time step, and the hidden state of the previous time step; determining the output value of the reset gate of the preset deep learning model according to the second weight matrix of the update gate, the bias item of the update gate, the data sample of the current time step, and the hidden state of the previous time step; determining the candidate hidden state of the current time step according to the output value of the reset gate of the preset deep learning model, the data sample of the current time step, and the hidden state of the previous time step; determining the hidden state of the next time step according to the candidate hidden state of the current time step, the output sample data of the update gate, and the hidden state of the previous time step; determining the timing characteristics according to the hidden state of the next time step and the preset deep learning model.

[0106] Specifically, the present invention can determine the output sample data of the update gate by the following formula (3):

[0107]

[0108] Among them, z t To update the output sample data of the gate, is the sigmoid function, W z is the first weight matrix of the update gate, h t-1 is the hidden state of the previous time step, x t is the data sample of the current time step, b z is the bias term for the update gate.

[0109] The present invention can determine the output value of the reset gate by the following formula (4):

[0110]

[0111] Among them, r t To reset the output value of the gate, is the sigmoid function, W r is the second weight matrix of the update gate, h t-1 is the hidden state of the previous time step, x t is the data sample of the current time step, b z is the bias term for the update gate.

[0112] The present invention can determine the candidate hidden state of the current time step by the following formula (5):

[0113]

[0114] in, is the candidate hidden state of the current time step, W is the network training weight matrix of the preset deep learning model, r t is the output value of the reset gate, h t-1 is the hidden state of the previous time step, x t is the data sample at the current time step.

[0115] The present invention can determine the hidden state of the next time step by the following formula (6):

[0116]

[0117] Among them, h t is the hidden state of the next time step, z t Update the output sample data of the gate, h t-1 is the hidden state of the previous time step, is the candidate hidden state of the current time step. After presetting the deep learning model, the time series features can be obtained.

[0118] Step 103: Determine a feature data set according to the time series features and a preset time series convolutional neural network.

[0119] In the embodiment of the present invention, after determining the time series features, a feature data set can be determined based on the time series features and a preset time series convolutional neural network. Specifically, the feature data set can be obtained by the following formula (7):

[0120] x F =TCN(y t )Formula (7)

[0121] Among them, x F is the feature data set, TCN is the preset temporal convolutional neural network, y t In the present invention, the feature data set may also represent the output data samples of the preset time series convolutional neural network.

[0122] In the present invention, the preset temporal convolutional neural network can be TCN (Temporal Convolutional Network), which is a deep learning architecture specifically used to process sequence data. TCN replaces the traditional recurrent neural network (RNN) with convolution operation, solving the efficiency and gradient vanishing problems of RNN when processing long sequence data.

[0123] Step 104: training a preset model according to the feature data set to obtain a target prediction model.

[0124] In the embodiment of the present invention, after the feature data set is determined, a preset model may be trained according to the feature data set to obtain a target prediction model.

[0125] In one embodiment, training a preset model according to a feature data set to obtain a target prediction model may include: aggregating the feature data set into a four-dimensional stereo feature according to a preset dimensional aggregation function, a preset training weight, and a preset training bias; and training a preset model according to the four-dimensional stereo feature to obtain a target prediction model.

[0126] In the present invention, dimension aggregation can be used to aggregate feature data sets into four-dimensional stereo features. Specifically, the four-dimensional stereo features can be obtained by the following formula (8):

[0127] x=ΠΓ(K*x F +b F )Formula (8)

[0128] Among them, x is the four-dimensional stereo feature, Γ is the preset dimension aggregation function, K is the preset training weight, x F is the feature data set, b F Preset training bias.

[0129] In one embodiment, a preset model is trained according to four-dimensional stereo features to obtain a target prediction model, which may include: determining sample transposed data information, positive sample information and negative sample information according to the four-dimensional stereo features; and training a preset model according to the sample transposed data information, positive sample information and negative sample information to obtain a target prediction model.

[0130] In the present invention, the preset model may include an encoder and a decoder. For the four-dimensional stereo features, a self-supervised learning method based on contrastive learning may be used to train the encoder. The encoder is trained by comparing the positive sample and the sample after the decoder. The contrastive learning maximizes the similarity between different views to learn the feature representation. The encoding result of the encoder can be obtained by the following formula (9):

[0131]

[0132] Among them, Z is the encoding result of the encoder, x is the four-dimensional stereo feature, that is, the training sample of the encoder, For the encoder.

[0133] When training the preset model, the four-dimensional stereo features may include a training set and a test set. The training set is used for training the preset model. The four-dimensional stereo features may be divided into a training set and a test set according to a preset ratio. The loss function for training the preset model is the InfoNCE function. The similarity maximization of positive samples may be used. By maximizing the similarity between the target sample and the positive sample, the preset model can capture the potential relationship between them. The target sample may be a normalized data set. The similarity minimization of negative samples is used. By minimizing the similarity between the target sample and the negative sample, it is ensured that the model can distinguish between the target sample and samples that are not related to it. Different data features are generated through data enhancement (such as time window sliding, noise addition), and the training model maximizes the similarity between different features of the same time series. Through the following formula (10), the similarity between different features of the same time series can be maximized, thereby realizing the training of the preset model:

[0134]

[0135] Among them, Loss N To maximize the similarity between different features of the same time series, E X It represents the product of the inverse of the number of samples and the cumulative function. The number of samples can be the number of four-dimensional features, log is the logarithmic function, exp is the exponential function, f(x) T is the sample transposed data information, f(x + ) is the positive sample information, f(x j ) is the negative sample information.

[0136] The present invention can use a stochastic gradient descent (SGD) optimization function in model training. SGD updates model parameters by using only one or a small number of samples in each iteration, thereby accelerating the training process and avoiding falling into a local optimal solution to a certain extent.

[0137] The present invention uses a preset deep learning model and a preset time series convolutional neural network to extract data features, and aggregates all types of data into four-dimensional spatial stereo data. Subsequently, contrast-based self-supervised learning is used to compare positive samples with samples after the encoder and decoder to train an encoder with strong representation capabilities to complete subsequent prediction tasks.

[0138] Step 105, acquiring real-time data of the system, and determining a scheduling strategy for data migration of the system according to the real-time data and the target prediction model.

[0139] In an embodiment of the present invention, after obtaining the target prediction model, real-time data of the system can be acquired in real time. The real-time data may include system indicator data and task data. After obtaining the real-time data, standardization and time window sliding processing can be performed to make it meet the input requirements of the target prediction model. The target prediction model is then used to perform predictions and output prediction results. The prediction results can reflect the system load and data change trends, and the system's data migration scheduling strategy can be determined based on the prediction results.

[0140] Specifically, after obtaining the prediction results, the prediction results can be processed and analyzed to make corresponding scheduling solutions. When analyzing the prediction results, the threshold can be used for judgment. The threshold range of system load and data changes can be set. By comparing the prediction results with the threshold, it can be judged whether the system is in a state of high load or drastic data changes. When analyzing the trend, the trend of the prediction results can be analyzed by the sliding average method to judge the future direction of system load and data changes and obtain trend information. Then, according to the results of processing and analysis, the scheduling strategy of data migration is dynamically adjusted. Among them, the scheduling strategy can include task priority, batch size and execution frequency.

[0141] Task priority adjustment: Dynamically adjust the execution order of tasks according to system load and business needs to ensure that high-priority tasks are given priority. The specific priorities are divided into 10 levels. For time periods with lower predicted loads, schedule high-priority tasks to ensure their timely completion. For time periods with higher predicted loads, schedule low-priority tasks to avoid system overload. High-priority tasks (7-10): tasks that have a greater impact on the business, such as synchronization and migration of critical data, and tasks with high real-time requirements. Medium-priority tasks (4-6): tasks that have a certain impact on the business, such as regular backups and batch data processing. Low-priority tasks (0-3): tasks that have less impact on the business, such as log archiving and migration of non-critical data. The priority adjustment strategy is:

[0142] Load balancing: When the system load is low, high-priority tasks are prioritized to ensure that critical tasks are completed in a timely manner. When the system load is high, low-priority tasks are appropriately postponed to avoid system overload.

[0143] Dynamic adjustment: Dynamically adjust the priority of tasks based on real-time predicted system load and data change trends. When it is predicted that the system load will increase, high-priority tasks will be processed in advance and low-priority tasks will be postponed.

[0144] Batch size adjustment: Dynamically adjust the batch size of data migration tasks based on the predicted system load. The batch size is within a specific range, such as 0-20; 20-30, etc. For periods of low load, the batch size can be increased to improve data migration efficiency. For periods of high load, the batch size can be reduced to balance the system load. The batch size is defined as large batches (above 1000) for situations with low system load and sufficient network bandwidth, improving the efficiency of data migration and synchronization; medium batches (100-1000) for situations with relatively flat system load and unlimited network bandwidth; small batches (less than 100) for situations with high system load and limited network bandwidth, reducing system pressure and preventing overload.

[0145] Execution frequency adjustment: Dynamically adjust the execution frequency of tasks based on the predicted system load and data change frequency. The specific execution frequency is divided into 1-10 frequency levels. For time periods with high data change frequency, the task execution frequency can be increased to ensure data consistency. For time periods with high load, the task execution frequency can be reduced to reduce system pressure.

[0146] In order to better understand the embodiments of the present invention, Figure 2 As shown, a schematic diagram of a data migration framework provided by an embodiment of the present invention is shown. Sample data is collected, and the sample data includes system indicators, task data, and historical records. The sample data is preprocessed to obtain a preprocessed data set. The data preprocessing includes: data cleaning, data enhancement, data conversion, and data aggregation. Then, according to the preset deep learning model, the preprocessed data set, and the preset time series convolutional neural network, the preset deep learning model is CRU, and the preset time series convolutional neural network is TCN, the data is processed again, for example, the data is aggregated into a four-dimensional stereo feature, and then the four-dimensional stereo feature is input into the encoder in the preset model. The encoder outputs the change result, and inputs the encoded result to the decoder. The decoder outputs the sample, and the encoder is trained by comparing the samples after the decoder. The probability value of each category, that is, Real or Fake, can be calculated by the SoftMax function, so as to train the encoder of the preset model and obtain the target prediction model. Then the real-time data of the system is obtained and the real-time data is input into the target prediction model. The encoder of the target prediction model can output the prediction result and determine the scheduling strategy of the system's data migration. The scheduling strategy can include task priority, batch size and execution frequency.

[0147] The present invention is a real-time example, collecting sample data, and preprocessing the sample data to obtain a preprocessed data set; according to the preset deep learning model and the preprocessed data set, determining the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model, and determining the timing characteristics according to the output sample data of the update gate and the output value of the reset gate; determining the characteristic data set according to the timing characteristics and the preset timing convolutional neural network; training the preset model according to the characteristic data set to obtain the target prediction model; acquiring the real-time data of the system, and determining the scheduling strategy of the data migration of the system according to the real-time data and the target prediction model. Thus, after preprocessing the sample data, the preset deep learning model and the preset timing convolutional neural network are used to determine the data set for training the preset model, so that when the target prediction model obtained by training is used to perform data migration prediction, the efficiency and reliability of its prediction are improved.

[0148] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0149] Reference Figure 3 , shows a structural block diagram of a data migration device provided by an embodiment of the present invention, and the device may specifically include the following modules:

[0150] The preprocessing module 301 is used to collect sample data and preprocess the sample data to obtain a preprocessed data set;

[0151] A first determination module 302 is used to determine the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model according to the preset deep learning model and the preprocessed data set, and determine the timing feature according to the output sample data of the update gate and the output value of the reset gate;

[0152] A second determination module 303 is used to determine a feature data set according to the time series feature and a preset time series convolutional neural network;

[0153] A training module 304 is used to train a preset model according to the feature data set to obtain a target prediction model;

[0154] The third determination module 305 is used to obtain real-time data of the system and determine a scheduling strategy for data migration of the system according to the real-time data and the target prediction model.

[0155] In one embodiment, the first determining module 302 includes:

[0156] A first determination submodule is used to determine the data sample of the current time step and the hidden state of the previous time step from the preprocessed data set by using a preset deep learning model;

[0157] A second determination submodule, used to determine the output sample data of the update gate according to the first weight matrix of the update gate of the preset deep learning model, the bias term of the update gate, the data sample of the current time step, and the hidden state of the previous time step;

[0158] A third determination submodule, configured to determine an output value of a reset gate of the preset deep learning model according to a second weight matrix of the update gate, a bias term of the update gate, a data sample of the current time step, and a hidden state of the previous time step;

[0159] A fourth determination submodule, configured to determine a candidate hidden state for the current time step according to an output value of a reset gate of the preset deep learning model, a data sample of the current time step, and a hidden state of the previous time step;

[0160] a fifth determination submodule, configured to determine a hidden state for a next time step based on the candidate hidden state for the current time step, the output sample data of the update gate, and the hidden state for the previous time step;

[0161] The sixth determination submodule is used to determine the timing characteristics according to the hidden state of the next time step and the preset deep learning model.

[0162] In one embodiment, the training module 304 includes:

[0163] A first aggregation submodule, configured to aggregate the feature data set into a four-dimensional stereo feature according to a preset dimensional aggregation function, a preset training weight, and a preset training bias;

[0164] The training submodule is used to train the preset model according to the four-dimensional stereo features to obtain a target prediction model.

[0165] In one embodiment, the training submodule includes:

[0166] A first determining unit, used to determine sample transposed data information, positive sample information and negative sample information according to the four-dimensional stereo feature;

[0167] The training unit is used to train a preset model according to the sample transposed data information, the positive sample information and the negative sample information to obtain a target prediction model.

[0168] In one embodiment, the preprocessing module 301 includes:

[0169] A seventh determination submodule, used to determine missing values ​​in the sample data;

[0170] A filling submodule is used to fill the missing values ​​by using a mean filling method to obtain a cleaned data set;

[0171] An enhancement submodule, used to perform data enhancement processing on the cleaned data set to obtain an enhanced data set;

[0172] A conversion submodule, used to perform data conversion processing on the enhanced data set to obtain a converted data set;

[0173] The second aggregation submodule is used to perform data aggregation processing on the converted data set to obtain a preprocessed data set.

[0174] In one embodiment, the conversion submodule includes:

[0175] A second determining unit, used to determine a maximum value and a minimum value of data in the enhanced data set;

[0176] A normalization unit, used to obtain a normalized data set according to the enhanced data set, the data maximum value and the data minimum value;

[0177] A generating unit, used to determine the time series data in the enhanced data set, and process the time series data in a time sliding window manner to generate input and output sample data;

[0178] The obtaining unit is used to obtain a converted data set according to the normalized data set and the input and output sample data.

[0179] In one embodiment, the enhancer module comprises:

[0180] A setting unit, used for setting a strategy search space;

[0181] A third determining unit, configured to determine a target enhancement strategy from the strategy search space by using a preset search algorithm;

[0182] The enhancement unit is used to perform data enhancement processing on the cleaned data set through the target enhancement strategy to obtain an enhanced data set.

[0183] In an embodiment of the present invention, a preprocessing module is used to collect sample data and preprocess the sample data to obtain a preprocessed data set; a first determination module is used to determine the output sample data of the update gate of the preset deep learning model and the output value of the reset gate of the preset deep learning model according to the preset deep learning model and the preprocessed data set, and determine the timing characteristics according to the output sample data of the update gate and the output value of the reset gate; a second determination module is used to determine the feature data set according to the timing characteristics and the preset timing convolutional neural network; a training module is used to train the preset model according to the feature data set to obtain a target prediction model; a third determination module is used to obtain the real-time data of the system, and determine the scheduling strategy of the data migration of the system according to the real-time data and the target prediction model. Thus, after the sample data is preprocessed, the data set used to train the preset model is determined by the preset deep learning model and the preset timing convolutional neural network, so that the target prediction model obtained by training can improve its prediction efficiency and reliability when performing data migration prediction.

[0184] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] An embodiment of the present invention further provides an electronic device, including:

[0186] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned data migration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0187] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned data migration method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0188] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0189] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0190] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0193] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0194] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0195] The data migration method, the data migration device, the electronic device and the computer-readable storage medium provided by the present invention are introduced in detail above. The principles and implementation methods of the present invention are explained in this article by using specific examples. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for the general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and the scope of application. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data migration method, characterized in that: The method comprises: Collecting sample data, and preprocessing the sample data to obtain a preprocessed data set; Determine, according to a preset deep learning model and the preprocessed data set, output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model, and determine a timing feature according to the output sample data of the update gate and the output value of the reset gate; Determine a feature data set according to the time series features and a preset time series convolutional neural network; Training a preset model according to the feature data set to obtain a target prediction model; Real-time data of the system is acquired, and a scheduling strategy for data migration of the system is determined based on the real-time data and the target prediction model.

2. The data migration method according to claim 1, characterized in that: The step of determining output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model according to the preset deep learning model and the preprocessed data set, and determining the timing characteristics according to the output sample data of the update gate and the timing characteristics, including: Determine the data sample of the current time step and the hidden state of the previous time step from the preprocessed data set by using a preset deep learning model; Determine output sample data of the update gate according to a first weight matrix of the update gate of the preset deep learning model, a bias term of the update gate, a data sample of the current time step, and a hidden state of the previous time step; Determining an output value of a reset gate of the preset deep learning model according to a second weight matrix of the update gate, a bias term of the update gate, a data sample of the current time step, and a hidden state of the previous time step; Determine a candidate hidden state for the current time step according to an output value of a reset gate of the preset deep learning model, a data sample of the current time step, and a hidden state of the previous time step; Determine the hidden state of the next time step according to the candidate hidden state of the current time step, the output sample data of the update gate, and the hidden state of the previous time step; Determine the timing characteristics according to the hidden state of the next time step and the preset deep learning model.

3. The data migration method according to claim 1, characterized in that: The step of training a preset model according to the feature data set to obtain a target prediction model includes: Aggregating the feature data set into a four-dimensional stereo feature according to a preset dimensional aggregation function, a preset training weight, and a preset training bias; According to the four-dimensional stereo features, the preset model is trained to obtain a target prediction model.

4. The data migration method according to claim 3, characterized in that: The method of training a preset model according to the four-dimensional stereo features to obtain a target prediction model includes: Determining sample transposed data information, positive sample information, and negative sample information according to the four-dimensional stereo features; The preset model is trained according to the sample transposed data information, the positive sample information and the negative sample information to obtain a target prediction model.

5. The data migration method according to claim 1, characterized in that: The preprocessing of the sample data to obtain a preprocessed data set includes: determining missing values ​​in the sample data; The missing values ​​are filled using the mean filling method to obtain a cleaned data set; Performing data enhancement processing on the cleaned data set to obtain an enhanced data set; Performing data conversion processing on the enhanced data set to obtain a converted data set; Data aggregation processing is performed on the converted data set to obtain a preprocessed data set.

6. The data migration method according to claim 5, characterized in that: The performing data conversion processing on the enhanced data set to obtain a converted data set includes: Determine the maximum value and the minimum value of the data in the enhanced data set; Obtaining a normalized data set according to the enhanced data set, the data maximum value, and the data minimum value; Determine the time series data in the enhanced data set, and process the time series data using a time sliding window method to generate input and output sample data; A converted data set is obtained according to the normalized data set and the input and output sample data.

7. The data migration method according to claim 5, characterized in that: The step of performing data enhancement processing on the cleaned data set to obtain an enhanced data set includes: Set the strategy search space; Determining a target enhancement strategy from the strategy search space by using a preset search algorithm; The target enhancement strategy is used to perform data enhancement processing on the cleaned data set to obtain an enhanced data set.

8. A data migration device, characterized in that: The device comprises: A preprocessing module is used to collect sample data and preprocess the sample data to obtain a preprocessed data set; A first determination module is used to determine, according to a preset deep learning model and the preprocessed data set, output sample data of an update gate of the preset deep learning model and an output value of a reset gate of the preset deep learning model, and determine a timing feature according to the output sample data of the update gate and the output value of the reset gate; A second determination module is used to determine a feature data set according to the time series features and a preset time series convolutional neural network; A training module, used to train a preset model according to the feature data set to obtain a target prediction model; The third determination module is used to obtain real-time data of the system and determine a scheduling strategy for data migration of the system based on the real-time data and the target prediction model.

9. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 7 are implemented.

Images