Multi-data-source data migration method and system

Abstract

The invention relates to the technical field of electric data migration, in particular to a multi-data-source data migration method and system, and the method comprises the steps: confirming an outflow processing platform and a plurality of electric data streams, obtaining a plurality of connected data source connectors, obtaining an electric data stream data packet set, obtaining a normal node set and an emigration node set, and obtaining a node topological structure diagram, identifying the node topological structure diagram to obtain an identified topological structure diagram, sequentially extracting emigration nodes from the emigration node set, confirming a target emigration position, obtaining a candidate response node set, obtaining an optimal response node according to the emigration nodes and the candidate response node set, and receiving a data migration instruction. And performing data migration on the emigration node according to the optimal response node and the data migration instruction to obtain migrated nodes, summarizing the migrated nodes to obtain a migrated node set, and completing the data migration method of the multiple data sources based on the migrated node set. According to the invention, the efficiency and timeliness of data migration can be improved.

Description

Technical Field

[0001] This invention relates to the field of electrical data migration technology, and in particular to a data migration method and system for multiple data sources. Background Technology

[0002] Multiple data sources refer to data originating from multiple different locations. Data migration refers to the process of moving data from one location or system to another.

[0003] Despite the increasing demand for data migration, existing data migration methods have several limitations. Traditional methods typically rely on manual operations or simple scripting tools, which are inefficient and error-prone when handling multiple data sources. Secondly, existing methods often fail to adequately consider resource management, unable to dynamically adjust resource allocation during the migration process, leading to resource waste or system overload. Therefore, improving the efficiency and timeliness of data migration is an urgent technical challenge. Summary of the Invention

[0004] This invention provides a data migration method and a computer-readable storage medium from multiple data sources, the main purpose of which is to improve the efficiency and timeliness of data migration.

[0005] To achieve the above objectives, the present invention provides a data migration method for multiple data sources, comprising: The stream processing platform and multiple electrical data streams were identified. The stream processing platform includes multiple data source connectors. Based on multiple data source connectors and multiple electrical data streams, acquire multiple connected data source connectors, and acquire electrical data stream data sets based on multiple connected data source connectors; Based on the electrical data stream data set, the normal node set and the migrated node set are obtained, and the node topology diagram is obtained. The node topology diagram is then marked using the normal node set and the migrated node set to obtain the marked topology diagram. Extract the outgoing nodes sequentially from the set of outgoing nodes. Based on the extracted outgoing nodes, identify the target outgoing location in the topology diagram. Based on the target outgoing location, obtain the set of candidate receiving nodes from the topology diagram. The best receiving node is obtained based on the set of outgoing nodes and candidate receiving nodes. Data migration instructions are received, and data migration is performed on the outgoing nodes based on the best receiving node and data migration instructions to obtain the migrated nodes. The method involves summarizing the migrated nodes to obtain a set of migrated nodes, and then using this set to complete the data migration from multiple data sources.

[0006] Optionally, the step of acquiring electrical data stream data packets based on multiple connected data source connectors includes: Perform the following operation for each of the multiple connected data source connectors: The pre-built load monitor is used to monitor the connected data source connectors to obtain the electrical data flow rate sequence. The electrical data flow ID is obtained according to the connected data source connector. The electrical data flow rate sequence includes multiple electrical data flow rates. The electrical data stream ID and electrical data stream speed sequence are packaged to obtain electrical data stream data packets. The electrical data stream data packets are then aggregated to obtain an electrical data stream data packet set. The electrical data stream data packet set includes multiple electrical data stream data packets, and each electrical data stream data packet, connected data source connector, and electrical data stream corresponds one-to-one.

[0007] Optionally, obtaining the normal node set and the emigrating node set based on the electrical data stream data packet set includes: Extract electrical data stream packets sequentially from the electrical data stream packet set, and perform the following operations on each extracted electrical data stream packet: The stationarity of the electrical data rate sequence in the electrical data stream data packet is tested to obtain the detection result, which is either stable or unstable. If the detection result is unstable, a differential velocity sequence is obtained based on the electrical data flow velocity sequence. The differential velocity sequence is used as the electrical data flow velocity sequence, and the step of performing a stationarity test on the electrical data flow velocity sequence in the electrical data flow data packet is returned until the detection result is stable. The differential velocity sequence corresponding to the stable detection result is used as the stationary data flow velocity sequence, and the corresponding difference order is determined based on the stationary data flow velocity sequence. The optimal prediction model is obtained based on the stationary data flow velocity sequence and the difference order. If the detection result is stable, the optimal prediction model is obtained based on the electrical data flow rate sequence; The optimal prediction model is used to predict the electrical data stream corresponding to the extracted electrical data stream data packets to obtain a set of predicted resource usage parameters, which includes: predicted CPU utilization, predicted memory utilization, predicted network bandwidth utilization, and predicted disk I / O utilization. The pre-built resource determination module is used to determine the predicted resource occupancy parameter set to obtain either the emigrating node or the normal node. By summarizing the normal nodes and the outgoing nodes respectively, we obtain the normal node set and the outgoing node set.

[0008] Optionally, obtaining the differential velocity sequence based on the electrical data velocity sequence includes: Extract the electrical data rate sequentially from the electrical data rate sequence, and perform the following operations on each extracted electrical data rate: Based on the extracted electrical data flow rate, adjacent electrical data flow rates are obtained from the electrical data flow rate sequence, wherein adjacent electrical data flow rates are adjacent to each other in the electrical data flow rate sequence and lag behind the electrical data flow rate. The electrical data flow rate difference is calculated based on the electrical data flow rate and adjacent electrical data flow rates. The electrical data flow rate difference is the value obtained by subtracting the adjacent electrical data flow rates from the electrical data flow rate. By summing the differences in electrical data flow rates, a differential flow rate sequence is obtained.

[0009] Optionally, obtaining the optimal prediction model based on the stationary data flow rate sequence and the difference order includes: The candidate sets of autoregressive order and the candidate sets of shifted average order are identified. Autoregressive order candidate values ​​are extracted sequentially from the autoregressive order candidate set. Based on the extracted autoregressive order candidate values ​​and the shifted average order candidate set, an order combination group is obtained. The order combination group includes multiple order combinations, and each order combination includes one autoregressive order candidate value and one shifted average order candidate value. Summarize the order combination groups to obtain the order combination set. Extract order combinations from the order combination set one by one, and perform the following operations on each extracted order combination: The ARIMA model is constructed based on the difference order and the combination of orders. The ARIMA model is trained using a stationary data flow sequence to obtain the maximum likelihood function value. The total number of samples is obtained from the stationary data flow rate sequence, and the model BIC value is calculated based on the total number of samples, the maximum likelihood function value, and the difference order. The model BIC values ​​are summarized to obtain the model BIC value set. The order combination corresponding to the model with the smallest BIC value in the model BIC value set is taken as the optimal combination. The optimal prediction model is determined based on the optimal combination.

[0010] Optionally, the formula for calculating the BIC value of the model is as follows:

[0011] in, This represents the BIC value of the model. Indicates the candidate values ​​of the autoregression order. Indicates the candidate values ​​for the order of the translational mean. Represents the total number of samples. Represents the logarithmic function. This represents the maximum likelihood function value.

[0012] Optionally, the step of using a pre-built resource determination module to determine the predicted resource occupancy parameter set to obtain either a migrating node or a normal node includes: Based on the resource determination module, the lower and upper limits of resources are determined, and it is determined whether the predicted resource occupancy parameters in the predicted resource occupancy parameter set are all less than the lower limit of resources. If all predicted resource occupancy parameters in the predicted resource occupancy parameter set are less than the lower limit of resources, then the electrical data stream corresponding to the extracted electrical data stream data packet will be regarded as a resource surplus node. Otherwise, determine whether there is a predicted resource usage parameter in the set of predicted resource usage parameters that is greater than the resource limit. If there is a predicted resource occupancy parameter in the predicted resource occupancy parameter set that is greater than the resource limit, then the electrical data stream is regarded as a resource overload node. If there is no predicted resource usage parameter greater than the resource limit in the predicted resource usage parameter set, then the electrical data stream is treated as a normal node. Nodes with excess or overloaded resources will be designated as migration nodes.

[0013] Optionally, obtaining the optimal receiving node based on the migration node and the candidate receiving node set includes: If the node to be migrated is a node with surplus resources, then obtain the total amount of data to be migrated from the node with surplus resources, identify the set of surplus candidate response nodes from the set of candidate response nodes based on the node with surplus resources, and identify the best surplus response node from the set of surplus candidate response nodes based on the total amount of data to be migrated. If the node to be migrated is a resource overload node, then the node's predicted resource occupancy parameter set is determined based on the resource overload node, the maximum resource occupancy rate is determined based on the node's predicted resource occupancy parameter set, and the load reduction amount is calculated based on the maximum resource occupancy rate and the preset target resource occupancy rate. The load reduction amount is the value obtained by subtracting the target resource occupancy rate from the maximum resource occupancy rate. The required data flow rate is calculated based on the load reduction and maximum resource utilization. Overload migration data is obtained from the resource overload nodes based on the required data flow rate. An overload candidate response node set is obtained from the candidate response node set based on the resource overload nodes. The optimal overload response node is determined from the overload candidate response node set based on the overload migration data. The optimal excess response node or the optimal overload response node is taken as the optimal response node.

[0014] Optionally, the step of identifying the optimal surplus response node from the surplus candidate response node set based on the total amount of data to be migrated includes: Extract excess candidate response nodes sequentially from the excess candidate response node set, obtain the excess candidate node resource occupancy rate set based on the extracted excess candidate response nodes, and predict the response occupancy rate set based on the total amount of data to be migrated and the excess candidate node resource occupancy rate set; If there is a response occupancy rate greater than the preset resource limit in the response occupancy rate set, the excess candidate response nodes are removed from the excess candidate response node set to obtain an updated excess candidate response node set. The updated excess candidate response node set is used as the excess candidate response node set, and the step of extracting excess candidate response nodes from the excess candidate response node set is returned until all excess candidate response nodes in the excess candidate response node set have been extracted. If there is no response occupancy rate greater than the resource limit in the response occupancy rate set, then the excess candidate response nodes are regarded as valid response nodes; Summarize the effective support nodes to obtain the effective support node set, and obtain the best surplus support node based on the effective support node set.

[0015] To achieve the above objectives, the present invention also provides a data migration system for multiple data sources, comprising: The data source connection module is used to identify the stream processing platform and multiple electrical data streams. The stream processing platform includes: multiple data source connectors, multiple connected data source connectors based on the multiple data source connectors and multiple electrical data streams, and electrical data stream data packet sets based on the multiple connected data source connectors. The node identification module is used to obtain the normal node set and the migration node set based on the electrical data stream data set, obtain the node topology diagram, identify the node topology diagram using the normal node set and the migration node set to obtain the identified topology diagram, extract the migration nodes from the migration node set in sequence, identify the target migration location in the identified topology diagram based on the extracted migration nodes, and obtain the candidate receiving node set from the identified topology diagram based on the target migration location. The optimal receiving node confirmation module is used to obtain the optimal receiving node based on the migration node and the candidate receiving node set, receive data migration instructions, and perform data migration on the migration node according to the optimal receiving node and the data migration instructions to obtain the migrated node. The electrical data migration module is used to summarize migrated nodes, obtain a set of migrated nodes, and complete the data migration method for multiple data sources based on the set of migrated nodes.

[0016] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; The processor executes the instructions stored in the memory to implement the data migration method for multiple data sources described above.

[0017] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the data migration method for multiple data sources described above.

[0018] To address the problems described in the background art, this invention identifies a streaming processing platform and multiple electrical data streams. The streaming processing platform includes multiple data source connectors. This invention provides a stable operating environment for subsequent data processing and migration. The streaming processing platform can efficiently process real-time data streams, ensuring the timeliness and smoothness of the data migration process. Based on multiple data source connectors and multiple electrical data streams, multiple connected data source connectors are obtained. Based on these connected data source connectors, electrical data stream data sets are obtained. This invention connects to various data sources through data source connectors, ensuring the system can obtain data from different data sources, achieving the integration of multiple data sources, breaking down data silos, and enabling data from different sources to be processed on a unified platform. Based on the electrical data stream data set, a normal node set and a migration node set are obtained, and a node topology diagram is obtained. The node topology diagram is then identified using the normal node set and the migration node set, resulting in an identified topology diagram. This invention divides nodes into normal node sets and migration node sets, helping the system clearly understand which nodes' data needs to be migrated and which nodes remain in a normal state, facilitating targeted migration. This invention performs data migration operations on a clear and intuitive basis. The node topology diagram visually displays the relationships and connections between nodes. By identifying normal nodes and migration-out nodes, the migration target becomes clearer, facilitating monitoring and management by administrators. Migration nodes are extracted sequentially from the set of migration-out nodes. Based on these extracted nodes, the target migration location is confirmed in the topology diagram. A set of candidate receiving nodes is then obtained from the topology diagram based on the target migration location. By confirming the target migration location and obtaining the candidate receiving node set in the topology diagram, this invention finds suitable migration directions and potential receiving nodes for the migration-out nodes, improving the accuracy and efficiency of data migration. The optimal receiving node is selected based on the migration-out nodes and the candidate receiving node set. Data migration instructions are received, and data is migrated from the migration-out nodes according to the optimal receiving node and the data migration instructions, resulting in migrated nodes. This invention, by obtaining the optimal receiving node, can select the most suitable node to receive the data from the migration-out nodes, reducing data transmission latency and cost, and improving data migration performance. The migrated nodes are then summarized to obtain a migrated node set. Based on this migrated node set, a multi-data source data migration method is completed. Therefore, this invention can improve the efficiency and timeliness of data migration. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a data migration method for multiple data sources provided in an embodiment of the present invention; Figure 2 A functional block diagram of a multi-data source data migration system provided in an embodiment of the present invention; Figure 3This is a schematic diagram of the structure of an electronic device that implements the data migration method for multiple data sources according to an embodiment of the present invention.

[0020] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0023] This application provides a data migration method for multiple data sources. The execution entity of the multi-data-source data migration method includes, but is not limited to, at least one of the following: a server, a terminal, or an electronic device that can be configured to execute the method provided in this application. In other words, the multi-data-source data migration method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0024] Reference Figure 1 The diagram shown is a flowchart illustrating a data migration method for multiple data sources according to an embodiment of the present invention. In this embodiment, the data migration method for multiple data sources includes: S1. Confirm the stream processing platform and multiple electrical data streams, wherein the stream processing platform includes multiple data source connectors.

[0025] It should be explained that a stream processing platform is a software system used for real-time processing and analysis of streaming data, capable of receiving, processing, and analyzing continuous data streams from multiple data sources. Electrical data streams refer to data streams related to the power system, including smart meter readings, grid status data, and load forecast data. A data source connector is a component of the stream processing platform used to connect to external data sources and receive data. Each data source connector corresponds to a specific data source and is responsible for reading data from that data source and transmitting it to the stream processing platform.

[0026] S2. Based on multiple data source connectors and multiple electrical data streams, obtain multiple connected data source connectors, and obtain electrical data stream data sets based on multiple connected data source connectors.

[0027] Specifically, the acquisition of electrical data stream data packet sets based on multiple connected data source connectors includes: Perform the following operation for each of the multiple connected data source connectors: The pre-built load monitor is used to monitor the connected data source connectors to obtain the electrical data flow rate sequence. The electrical data flow ID is obtained according to the connected data source connector. The electrical data flow rate sequence includes multiple electrical data flow rates. The electrical data stream ID and electrical data stream speed sequence are packaged to obtain electrical data stream data packets. The electrical data stream data packets are then aggregated to obtain an electrical data stream data packet set. The electrical data stream data packet set includes multiple electrical data stream data packets, and each electrical data stream data packet, connected data source connector, and electrical data stream corresponds one-to-one.

[0028] It should be explained that a connected data source connector refers to a data source connector that has successfully connected to the data source corresponding to the electrical data stream. A load monitor is a tool used to monitor and measure the performance metrics of connected data source connectors in real time. For example, a performance metric is the data flow rate. An electrical data flow rate sequence refers to a sequence of recorded electrical data flow rates over a period of time. An electrical data flow ID is an identifier used to uniquely identify each electrical data flow, distinguishing different data sources when processing multiple electrical data flows. An electrical data flow data packet is a data structure that packages the electrical data flow ID and the electrical data flow rate sequence together. This invention's electrical data flow data packet is designed to facilitate data transmission and processing.

[0029] S3. Obtain the normal node set and the migrated node set based on the electrical data stream data set, obtain the node topology diagram, and use the normal node set and the migrated node set to identify the node topology diagram to obtain the identified topology diagram.

[0030] It should be explained that a node topology diagram is a graphical representation used to show the connections and organizational structure between various nodes in a system. The node topology diagram of this invention is a pre-constructed topology diagram based on the data source corresponding to each of multiple electrical data streams as nodes, and the connections between these data sources (such as network links). The identifier topology diagram is obtained by marking all nodes on the node topology diagram according to the normal node set and the migrated node set.

[0031] Specifically, the acquisition of the normal node set and the emigrating node set based on the electrical data stream data packet set includes: Extract electrical data stream packets sequentially from the electrical data stream packet set, and perform the following operations on each extracted electrical data stream packet: The stationarity of the electrical data rate sequence in the electrical data stream data packet is tested to obtain the detection result, which is either stable or unstable. If the detection result is unstable, a differential velocity sequence is obtained based on the electrical data flow velocity sequence. The differential velocity sequence is used as the electrical data flow velocity sequence, and the step of performing a stationarity test on the electrical data flow velocity sequence in the electrical data flow data packet is returned until the detection result is stable. The differential velocity sequence corresponding to the stable detection result is used as the stationary data flow velocity sequence, and the corresponding difference order is determined based on the stationary data flow velocity sequence. The optimal prediction model is obtained based on the stationary data flow velocity sequence and the difference order. If the detection result is stable, the optimal prediction model is obtained based on the electrical data flow rate sequence; The optimal prediction model is used to predict the electrical data stream corresponding to the extracted electrical data stream data packets to obtain a set of predicted resource usage parameters, which includes: predicted CPU utilization, predicted memory utilization, predicted network bandwidth utilization, and predicted disk I / O utilization. The pre-built resource determination module is used to determine the predicted resource occupancy parameter set to obtain either the emigrating node or the normal node. By summarizing the normal nodes and the outgoing nodes respectively, we obtain the normal node set and the outgoing node set.

[0032] It should be explained that electrical data flow rate refers to the flow rate value of electrical data flow at a specific point in the past. The stationarity test of the electrical data flow rate sequence in the electrical data flow data packets refers to using the Augmented Dickey-Fuller (ADF) test to test the stationarity of the electrical data flow rate sequence. The stationarity test checks whether time series data has stable statistical properties (such as mean, variance, etc.). The difference order refers to the number of differencing operations required to make the data in the electrical data flow rate sequence stationary. The method for obtaining the optimal prediction model based on the electrical data flow rate sequence is the same as the method for obtaining the optimal prediction model based on the stationary data flow rate sequence and the difference order, and will not be repeated here. The optimal prediction model refers to the ARIMA model that best fits the electrical data flow rate sequence and makes predictions after model selection and parameter optimization. Predicted CPU utilization refers to the proportion of CPU resources expected to be used by the electrical data flow at a future point in time or within a time window. A higher predicted CPU utilization indicates a heavier system load, potentially requiring more CPU resources to process tasks. Predicting CPU utilization allows for proactive resource allocation adjustments to avoid system overload or resource waste. Predicted memory utilization refers to the percentage of memory resources expected to be used by electrical data streams within a future timeframe or window. A higher predicted memory utilization indicates insufficient system memory, requiring more memory resources to process tasks. Predicted network bandwidth utilization refers to the percentage of network bandwidth resources expected to be used by electrical data streams within a future timeframe or window. A higher predicted network bandwidth utilization indicates high network traffic, requiring more bandwidth resources to handle data transmission. Proactive adjustments to network resource allocation are needed to avoid network congestion. Predicted disk I / O utilization refers to the percentage of disk I / O resources expected to be used by electrical data streams within a future timeframe or window. A higher predicted disk I / O utilization indicates frequent disk read / write operations, requiring more disk I / O resources to process data. Proactive adjustments to storage resource allocation are needed to avoid disk I / O bottlenecks. The resource assessment module is used to evaluate and determine the resource utilization of nodes. Its main function is to determine whether a node is suitable to continue carrying the current electrical data flow based on predicted resource utilization parameters (such as predicted CPU utilization, predicted memory utilization, predicted network bandwidth utilization, and predicted disk I / O utilization). The normal node set refers to the set of all normal nodes. The emigrating node set refers to the set of all emigrating nodes.

[0033] Specifically, the step of obtaining the differential velocity sequence based on the electrical data velocity sequence includes: Extract the electrical data rate sequentially from the electrical data rate sequence, and perform the following operations on each extracted electrical data rate: Based on the extracted electrical data flow rate, adjacent electrical data flow rates are obtained from the electrical data flow rate sequence, wherein adjacent electrical data flow rates are adjacent to each other in the electrical data flow rate sequence and lag behind the electrical data flow rate. The electrical data flow rate difference is calculated based on the electrical data flow rate and adjacent electrical data flow rates. The electrical data flow rate difference is the value obtained by subtracting the adjacent electrical data flow rates from the electrical data flow rate. By summing the differences in electrical data flow rates, a differential flow rate sequence is obtained.

[0034] It should be explained that the differential velocity sequence refers to a new sequence obtained by differentially analyzing a historical velocity data sequence. The difference in historical velocity data accurately reflects the change in data over adjacent time units (i.e., the instantaneous rate of change), effectively capturing the short-term fluctuation characteristics of the data and preparing for the elimination of trend components. The differential velocity sequence is a direct and effective input for subsequent stationarity tests (such as the ADF test) and the construction of ARIMA prediction models. By removing non-stationary factors from the historical velocity data sequence that interfere with prediction, it can significantly improve the accuracy and reliability of subsequent ARIMA models. For example, the electrical data flow rate sequence is {50, 52, 55, 53, 58}. Electrical data flow rate 50 is extracted sequentially from the sequence. Then, based on the extracted electrical data flow rate 50, the adjacent electrical data flow rate 52 is obtained from the sequence. The difference between the extracted electrical data flow rate 50 and the adjacent electrical data flow rate 52 is calculated to be -2. The adjacent electrical data flow rate 52 is taken as the extracted electrical data flow rate. Next, based on the extracted electrical data flow rate 52, the adjacent electrical data flow rate 55 is obtained from the sequence. The difference between the extracted electrical data flow rate 52 and the adjacent electrical data flow rate 55 is calculated to be -3. The adjacent electrical data flow rate of 55 is taken as the extracted electrical data flow rate. Based on the extracted electrical data flow rate of 55, the adjacent electrical data flow rate of 53 is obtained from the electrical data flow rate sequence. The electrical data flow rate difference of 2 is calculated based on the extracted electrical data flow rate of 55 and the adjacent electrical data flow rate of 53. The adjacent electrical data flow rate of 53 is taken as the extracted electrical data flow rate. Based on the extracted electrical data flow rate of 53, the adjacent electrical data flow rate of 58 is obtained from the electrical data flow rate sequence. The electrical data flow rate difference of -5 is calculated based on the extracted electrical data flow rate of 53 and the adjacent electrical data flow rate of 58. By summing up all the above electrical data flow rate differences, the differential flow rate sequence is obtained as {-2, -3, 2, -5}.

[0035] Specifically, obtaining the optimal prediction model based on the stationary data flow rate sequence and the difference order includes: The candidate sets of autoregressive order and the candidate sets of shifted average order are identified. Autoregressive order candidate values ​​are extracted sequentially from the autoregressive order candidate set. Based on the extracted autoregressive order candidate values ​​and the shifted average order candidate set, an order combination group is obtained. The order combination group includes multiple order combinations, and each order combination includes one autoregressive order candidate value and one shifted average order candidate value. Summarize the order combination groups to obtain the order combination set. Extract order combinations from the order combination set one by one, and perform the following operations on each extracted order combination: The ARIMA model is constructed based on the difference order and the combination of orders. The ARIMA model is trained using a stationary data flow sequence to obtain the maximum likelihood function value. The total number of samples is obtained from the stationary data flow rate sequence, and the model BIC value is calculated based on the total number of samples, the maximum likelihood function value, and the difference order. The model BIC values ​​are summarized to obtain the model BIC value set. The order combination corresponding to the model with the smallest BIC value in the model BIC value set is taken as the optimal combination. The optimal prediction model is determined based on the optimal combination.

[0036] It should be explained that the candidate set of autoregressive order values ​​is a set of possible autoregressive order values. The candidate set of shifted average order values ​​refers to a set of possible shifted average order values. The order combination set is a set composed of all possible combinations of autoregressive order candidate values ​​and shifted average order candidate values. Obtaining the order combination set based on the extracted autoregressive order candidate values ​​and shifted average order candidate value sets means combining the extracted autoregressive order candidate values ​​with each shifted average order candidate value in the shifted average order candidate value set to obtain the order combination set. An ARIMA model is a statistical model used for time series forecasting. The maximum likelihood function value is an indicator of how well an ARIMA model fits the data. In an ARIMA model, the maximum likelihood function value is obtained through maximum likelihood estimation. The total number of samples refers to the number of stationary data flows in a stationary data flow sequence. The model BIC value set refers to the set of model BIC values ​​corresponding to all order combinations in the order combination set. The optimal prediction model refers to the ARIMA model composed of the optimal combination and the difference order.

[0037] In detail, the formula for calculating the BIC value of the model is as follows:

[0038] in, This represents the BIC value of the model. Indicates the candidate values ​​of the autoregression order. Indicates the candidate values ​​for the order of the translational mean. Represents the total number of samples. Represents the logarithmic function. This represents the maximum likelihood function value.

[0039] It needs to be explained that, The larger the BIC value, the more complex the ARIMA model and the more prone it is to overfitting. The core purpose of the BIC calculation formula is to select the optimal autoregressive and moving average orders for the ARIMA model, seeking the best balance between goodness of fit and model complexity through mathematical quantification. This formula... The program rewards models that accurately fit historical data, ensuring they capture core patterns, while also... This approach penalizes complex models with too many parameters, forcing them to pursue simplicity and avoid overlearning on noise. It automatically selects the optimal model that can fully capture data patterns while being simple enough to avoid overfitting, providing a reliable and efficient foundation for subsequent load forecasting.

[0040] In detail, the step of using a pre-built resource determination module to determine the predicted resource occupancy parameter set to obtain either a migrating node or a normal node includes: Based on the resource determination module, the lower and upper limits of resources are determined, and it is determined whether the predicted resource occupancy parameters in the predicted resource occupancy parameter set are all less than the lower limit of resources. If all predicted resource occupancy parameters in the predicted resource occupancy parameter set are less than the lower limit of resources, then the electrical data stream corresponding to the extracted electrical data stream data packet will be regarded as a resource surplus node. Otherwise, determine whether there is a predicted resource usage parameter in the set of predicted resource usage parameters that is greater than the resource limit. If there is a predicted resource occupancy parameter in the predicted resource occupancy parameter set that is greater than the resource limit, then the electrical data stream is regarded as a resource overload node. If there is no predicted resource usage parameter greater than the resource limit in the predicted resource usage parameter set, then the electrical data stream is treated as a normal node. Nodes with excess or overloaded resources will be designated as migration nodes.

[0041] It should be explained that the resource lower limit refers to the minimum threshold set by the system for each resource (such as predicted CPU utilization, predicted memory utilization, predicted network bandwidth utilization, and predicted disk I / O utilization). When the predicted resource utilization parameter is lower than the resource lower limit, it indicates that the resource utilization of the electrical data stream is low, and there may be resource waste. The resource upper limit refers to the maximum threshold set by the system for each resource (such as predicted CPU utilization, predicted memory utilization, predicted network bandwidth utilization, and predicted disk I / O utilization). When the predicted resource utilization parameter is greater than this resource upper limit, it indicates that the resource utilization of the electrical data stream is too high, and there may be resource shortage. A resource surplus node is a node in which all predicted resource utilization parameters in the predicted resource utilization parameter set are lower than the resource lower limit. A resource overload node is a node in which at least one predicted resource utilization parameter in the predicted resource utilization parameter set exceeds the resource upper limit. A normal node is a node in which neither all predicted resource utilization parameters are lower than the resource lower limit nor any predicted resource utilization parameter exceeds the resource upper limit; this node can operate normally.

[0042] S4. Extract the outgoing nodes sequentially from the set of outgoing nodes. Based on the extracted outgoing nodes, identify the target outgoing location in the topology diagram. Based on the target outgoing location, obtain the set of candidate receiving nodes from the topology diagram.

[0043] It should be explained that the candidate receiving node set refers to the set of candidate receiving nodes in the topology diagram that may receive data from the migrating-out node during the data migration process. In this invention, the candidate receiving nodes are normal nodes. The target migration-out location refers to the position of the migrating-out node in the topology diagram.

[0044] S5. Obtain the best receiving node based on the set of outgoing nodes and candidate receiving nodes, receive the data migration instruction, and perform data migration on the outgoing nodes according to the best receiving node and the data migration instruction to obtain the migrated nodes.

[0045] Specifically, the step of obtaining the optimal receiving node based on the migration node and the set of candidate receiving nodes includes: If the node to be migrated is a node with surplus resources, then obtain the total amount of data to be migrated from the node with surplus resources, identify the set of surplus candidate response nodes from the set of candidate response nodes based on the node with surplus resources, and identify the best surplus response node from the set of surplus candidate response nodes based on the total amount of data to be migrated. If the node to be migrated is a resource overload node, then the node's predicted resource occupancy parameter set is determined based on the resource overload node, the maximum resource occupancy rate is determined based on the node's predicted resource occupancy parameter set, and the load reduction amount is calculated based on the maximum resource occupancy rate and the preset target resource occupancy rate. The load reduction amount is the value obtained by subtracting the target resource occupancy rate from the maximum resource occupancy rate. The required data flow rate is calculated based on the load reduction and maximum resource utilization. Overload migration data is obtained from the resource overload nodes based on the required data flow rate. An overload candidate response node set is obtained from the candidate response node set based on the resource overload nodes. The optimal overload response node is determined from the overload candidate response node set based on the overload migration data. The optimal excess response node or the optimal overload response node is taken as the optimal response node.

[0046] It should be explained that a data migration instruction is a system-issued command used to initiate and manage the data migration process. A migrated node refers to a node that has completed the data migration process. After data migration is complete, the data from the migrated node is successfully transferred to the optimal receiving node. The migrated node can then release its resources, while the optimal receiving node begins processing new data. The step of obtaining the total amount of data to be migrated from a resource-excess node is as follows: traverse all files in the resource-excess node, calculate the data volume corresponding to each file, and sum the data volumes of all files to obtain the total amount of data to be migrated. The set of excess candidate receiving nodes refers to the set of downstream nodes corresponding to the resource-excess node in the identifier topology diagram. The downstream node refers to the next node that receives and processes the output result of the current node. For example, node A... Node B Node C is the downstream node of node A, which is node B and node C.

[0047] Importantly, the method for identifying the node predicted resource occupancy parameter set based on resource overload nodes is the same as the method for obtaining predicted resource occupancy parameters based on electrical data flow, and will not be repeated here. The node predicted resource occupancy parameter set refers to the collection of node predicted resource occupancy parameters for the future resource occupancy of resource overload nodes. A node predicted resource occupancy parameter refers to the future resource occupancy rate of a resource overload node. The maximum resource occupancy rate refers to the highest value among all resource occupancy rates in the node predicted resource occupancy parameter set. The target resource occupancy rate is a pre-set occupancy rate designed to optimize resource utilization and avoid resource waste or overload. The step of calculating the required data flow rate value based on the required load reduction and the maximum resource occupancy rate is as follows: obtain the current data processing throughput; required data flow rate value = required load reduction × (current data processing throughput / maximum resource occupancy rate). The current data processing throughput refers to the rate at which the system or node can process the amount of data at the current point in time. The step of obtaining overload migration data from the resource-overloaded nodes based on the required data flow rate value refers to selecting data from several data partitions that have been processed the least recently or have the lightest load from the resource-overloaded nodes; this data constitutes the overload migration data. The method for obtaining the set of candidate overloaded response nodes from the candidate response node set based on resource-overloaded nodes is the same as the method for identifying the set of excess candidate response nodes from the candidate response node set based on resource-excess nodes, and will not be repeated here. The method for identifying the optimal overloaded response node from the set of candidate overloaded response nodes based on the overload migration data is the same as the method for identifying the optimal excess node from the set of excess candidate response nodes based on the total amount of data to be migrated, and will not be repeated here.

[0048] It should be noted that after all data in a resource-overloaded node is migrated out, the node is also deleted. This aims to fully recover the node's computational, memory, and energy consumption, achieving the most direct energy efficiency optimization. For resource-overloaded nodes, quantitative analysis is used to accurately calculate the minimum amount of data that needs to be migrated out, achieving the goal of alleviating the bottleneck with minimal migration. Simultaneously, the migrated data is safely diverted from the resource-overloaded node to a less loaded node (the optimal overload takeover node), realizing dynamic rebalancing of resources within the cluster. The resource corresponding to the maximum resource occupancy rate is the bottleneck resource currently restricting the performance of the resource-overloaded node and causing its overload. Therefore, the maximum resource occupancy rate is determined based on the node's predicted resource occupancy parameter set. If the migration amount is calculated for a non-bottleneck resource, then even if data is migrated out, the bottleneck pressure on the resource-overloaded node still exists, and the overload problem is not solved.

[0049] Specifically, the step of identifying the optimal surplus response node from the set of surplus candidate response nodes based on the total amount of data to be migrated includes: Extract excess candidate response nodes sequentially from the excess candidate response node set, obtain the excess candidate node resource occupancy rate set based on the extracted excess candidate response nodes, and predict the response occupancy rate set based on the total amount of data to be migrated and the excess candidate node resource occupancy rate set; If there is a response occupancy rate greater than the preset resource limit in the response occupancy rate set, the excess candidate response nodes are removed from the excess candidate response node set to obtain an updated excess candidate response node set. The updated excess candidate response node set is used as the excess candidate response node set, and the step of extracting excess candidate response nodes from the excess candidate response node set is returned until all excess candidate response nodes in the excess candidate response node set have been extracted. If there is no response occupancy rate greater than the resource limit in the response occupancy rate set, then the excess candidate response nodes are regarded as valid response nodes; Summarize the effective support nodes to obtain the effective support node set, and obtain the best surplus support node based on the effective support node set.

[0050] It should be explained that the surplus candidate node resource utilization set refers to the set of CPU utilization, memory utilization, network bandwidth utilization, and disk I / O utilization of the surplus candidate response nodes. Predicting the response utilization set based on the total amount of data to be migrated and the surplus candidate node resource utilization set means adding the utilization rate corresponding to the total amount of data to be migrated to the corresponding surplus candidate node resource utilization rate in the surplus candidate node resource utilization set, resulting in the set of response utilization rates. Response utilization rate refers to the resource utilization rate of the surplus candidate response nodes after considering the migrated data. Updating the surplus candidate response node set refers to the set of remaining surplus candidate response nodes after removing those whose response utilization rate exceeds a preset resource limit. A valid response node is a surplus candidate response node whose response utilization rate does not exceed a preset resource limit after the data migration. The valid response node set refers to the set of all valid response nodes. The step of obtaining the optimal surplus support node based on the effective support node set is as follows: Obtain the total support occupancy rate of each effective support node in the effective support node set, resulting in a total support occupancy rate set. The effective support node corresponding to the minimum total support occupancy rate in this set is taken as the optimal surplus support node. The total support occupancy rate refers to the sum of all support occupancy rates in the support occupancy rate set corresponding to the effective support node. The total support occupancy rate set is the set composed of all total support occupancy rates.

[0051] S6. Summarize the migrated nodes to obtain the migrated node set, and complete the data migration method for multiple data sources based on the migrated node set.

[0052] It is understood that the migrated node set refers to the collection of all migrated nodes. This invention summarizes the migrated nodes into a migrated node set, facilitating unified management and monitoring of the entire data migration process and enabling statistical analysis of the data migration results. The completion of data migration from multiple data sources based on the migrated node set marks the end of the entire data migration process, achieving the goal of data integration and migration between multiple data sources.

[0053] To address the problems described in the background art, this invention identifies a streaming processing platform and multiple electrical data streams. The streaming processing platform includes multiple data source connectors. This invention provides a stable operating environment for subsequent data processing and migration. The streaming processing platform can efficiently process real-time data streams, ensuring the timeliness and smoothness of the data migration process. Based on multiple data source connectors and multiple electrical data streams, multiple connected data source connectors are obtained. Based on these connected data source connectors, electrical data stream data sets are obtained. This invention connects to various data sources through data source connectors, ensuring the system can obtain data from different data sources, achieving the integration of multiple data sources, breaking down data silos, and enabling data from different sources to be processed on a unified platform. Based on the electrical data stream data set, a normal node set and a migration node set are obtained, and a node topology diagram is obtained. The node topology diagram is then identified using the normal node set and the migration node set, resulting in an identified topology diagram. This invention divides nodes into normal node sets and migration node sets, helping the system clearly understand which nodes' data needs to be migrated and which nodes remain in a normal state, facilitating targeted migration. This invention performs data migration operations on a clear and intuitive basis. The node topology diagram visually displays the relationships and connections between nodes. By identifying normal nodes and migration-out nodes, the migration target becomes clearer, facilitating monitoring and management by administrators. Migration nodes are extracted sequentially from the set of migration-out nodes. Based on these extracted nodes, the target migration location is confirmed in the topology diagram. A set of candidate receiving nodes is then obtained from the topology diagram based on the target migration location. By confirming the target migration location and obtaining the candidate receiving node set in the topology diagram, this invention finds suitable migration directions and potential receiving nodes for the migration-out nodes, improving the accuracy and efficiency of data migration. The optimal receiving node is selected based on the migration-out nodes and the candidate receiving node set. Data migration instructions are received, and data is migrated from the migration-out nodes according to the optimal receiving node and the data migration instructions, resulting in migrated nodes. This invention, by obtaining the optimal receiving node, can select the most suitable node to receive the data from the migration-out nodes, reducing data transmission latency and cost, and improving data migration performance. The migrated nodes are then summarized to obtain a migrated node set. Based on this migrated node set, a multi-data source data migration method is completed. Therefore, this invention can improve the efficiency and timeliness of data migration.

[0054] like Figure 2 The diagram shown is a functional block diagram of a multi-data source data migration system provided in an embodiment of the present invention.

[0055] The multi-data source data migration system 100 of the present invention can be installed in an electronic device. Depending on the functions implemented, the multi-data source data migration system 100 may include a data source connection module 101, a node identification module 102, an optimal response node confirmation module 103, and an electrical data migration module 104. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device. The data source connection module 101 is used to confirm the stream processing platform and multiple electrical data streams. The stream processing platform includes: multiple data source connectors, multiple connected data source connectors based on the multiple data source connectors and multiple electrical data streams, and electrical data stream data packet sets based on the multiple connected data source connectors. The node identification module 102 is used to obtain a normal node set and a migration node set based on the electrical data stream data set, obtain a node topology diagram, identify the node topology diagram using the normal node set and the migration node set to obtain an identified topology diagram, extract migration nodes sequentially from the migration node set, identify the target migration location in the identified topology diagram based on the extracted migration nodes, and obtain a candidate receiving node set from the identified topology diagram based on the target migration location. The optimal receiving node confirmation module 103 is used to obtain the optimal receiving node based on the migration node and the candidate receiving node set, receive the data migration instruction, and perform data migration on the migration node based on the optimal receiving node and the data migration instruction to obtain the migrated node. The electrical data migration module 104 is used to summarize the migrated nodes to obtain a set of migrated nodes, and to complete the data migration method for multiple data sources based on the set of migrated nodes.

[0056] In detail, the modules in the multi-data source data migration system 100 described in this embodiment of the invention employ the same methods as described above. Figure 1 The data migration method using multiple data sources described herein employs the same technical means and can produce the same technical effects, so it will not be elaborated here.

[0057] like Figure 3 The diagram shown is a structural schematic of an electronic device for implementing a data migration method for multiple data sources according to an embodiment of the present invention.

[0058] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a data migration method program for multiple data sources.

[0059] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for data migration methods from multiple data sources, but also to temporarily store data that has been output or will be output.

[0060] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (such as data migration methods from multiple data sources) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0061] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0062] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0063] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0064] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0065] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0066] The data migration method program for multiple data sources stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 10, can achieve the following: The stream processing platform and multiple electrical data streams were identified. The stream processing platform includes multiple data source connectors. Based on multiple data source connectors and multiple electrical data streams, acquire multiple connected data source connectors, and acquire electrical data stream data sets based on multiple connected data source connectors; Based on the electrical data stream data set, the normal node set and the migrated node set are obtained, and the node topology diagram is obtained. The node topology diagram is then marked using the normal node set and the migrated node set to obtain the marked topology diagram. Extract the outgoing nodes sequentially from the set of outgoing nodes. Based on the extracted outgoing nodes, identify the target outgoing location in the topology diagram. Based on the target outgoing location, obtain the set of candidate receiving nodes from the topology diagram. The best receiving node is obtained based on the set of outgoing nodes and candidate receiving nodes. Data migration instructions are received, and data migration is performed on the outgoing nodes based on the best receiving node and data migration instructions to obtain the migrated nodes. The method involves summarizing the migrated nodes to obtain a set of migrated nodes, and then using this set to complete the data migration from multiple data sources.

[0067] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0068] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0069] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: The stream processing platform and multiple electrical data streams were identified. The stream processing platform includes multiple data source connectors. Based on multiple data source connectors and multiple electrical data streams, acquire multiple connected data source connectors, and acquire electrical data stream data sets based on multiple connected data source connectors; Based on the electrical data stream data set, the normal node set and the migrated node set are obtained, and the node topology diagram is obtained. The node topology diagram is then marked using the normal node set and the migrated node set to obtain the marked topology diagram. Extract the outgoing nodes sequentially from the set of outgoing nodes. Based on the extracted outgoing nodes, identify the target outgoing location in the topology diagram. Based on the target outgoing location, obtain the set of candidate receiving nodes from the topology diagram. The best receiving node is obtained based on the set of outgoing nodes and candidate receiving nodes. Data migration instructions are received, and data migration is performed on the outgoing nodes based on the best receiving node and data migration instructions to obtain the migrated nodes. The method involves summarizing the migrated nodes to obtain a set of migrated nodes, and then using this set to complete the data migration from multiple data sources.

[0070] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0071] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0073] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A data migration method for multiple data sources, characterized in that, The method includes: The stream processing platform and multiple electrical data streams were identified. The stream processing platform includes multiple data source connectors. Based on multiple data source connectors and multiple electrical data streams, acquire multiple connected data source connectors, and acquire electrical data stream data sets based on multiple connected data source connectors; Based on the electrical data stream data set, the normal node set and the migrated node set are obtained, and the node topology diagram is obtained. The node topology diagram is then marked using the normal node set and the migrated node set to obtain the marked topology diagram. Extract the outgoing nodes sequentially from the set of outgoing nodes. Based on the extracted outgoing nodes, identify the target outgoing location in the topology diagram. Based on the target outgoing location, obtain the set of candidate receiving nodes from the topology diagram. The best receiving node is obtained based on the set of outgoing nodes and candidate receiving nodes. Data migration instructions are received, and data migration is performed on the outgoing nodes based on the best receiving node and data migration instructions to obtain the migrated nodes. The method involves summarizing the migrated nodes to obtain a set of migrated nodes, and then using this set to complete the data migration from multiple data sources.

2. The data migration method for multiple data sources as described in claim 1, characterized in that, The acquisition of electrical data stream data packets based on multiple connected data source connectors includes: Perform the following operation for each of the multiple connected data source connectors: The pre-built load monitor is used to monitor the connected data source connectors to obtain the electrical data flow rate sequence. The electrical data flow ID is obtained according to the connected data source connector. The electrical data flow rate sequence includes multiple electrical data flow rates. The electrical data stream ID and electrical data stream speed sequence are packaged to obtain electrical data stream data packets. The electrical data stream data packets are then aggregated to obtain an electrical data stream data packet set. The electrical data stream data packet set includes multiple electrical data stream data packets, and each electrical data stream data packet, connected data source connector, and electrical data stream corresponds one-to-one.

3. The data migration method for multiple data sources as described in claim 2, characterized in that, The process of obtaining the normal node set and the migration node set based on the electrical data stream data set includes: Extract electrical data stream packets sequentially from the electrical data stream packet set, and perform the following operations on each extracted electrical data stream packet: The stationarity of the electrical data rate sequence in the electrical data stream data packet is tested to obtain the detection result, which is either stable or unstable. If the detection result is unstable, a differential velocity sequence is obtained based on the electrical data flow velocity sequence. The differential velocity sequence is used as the electrical data flow velocity sequence, and the step of performing a stationarity test on the electrical data flow velocity sequence in the electrical data flow data packet is returned until the detection result is stable. The differential velocity sequence corresponding to the stable detection result is used as the stationary data flow velocity sequence, and the corresponding difference order is determined based on the stationary data flow velocity sequence. The optimal prediction model is obtained based on the stationary data flow velocity sequence and the difference order. If the detection result is stable, the optimal prediction model is obtained based on the electrical data flow rate sequence; The optimal prediction model is used to predict the electrical data stream corresponding to the extracted electrical data stream data packets to obtain a set of predicted resource usage parameters, which includes: predicted CPU utilization, predicted memory utilization, predicted network bandwidth utilization, and predicted disk I / O utilization. The pre-built resource determination module is used to determine the predicted resource occupancy parameter set to obtain either the emigrating node or the normal node. By summarizing the normal nodes and the outgoing nodes respectively, we obtain the normal node set and the outgoing node set.

4. The data migration method for multiple data sources as described in claim 3, characterized in that, The method of obtaining the differential velocity sequence based on the electrical data velocity sequence includes: Extract the electrical data rate sequentially from the electrical data rate sequence, and perform the following operations on each extracted electrical data rate: Based on the extracted electrical data flow rate, adjacent electrical data flow rates are obtained from the electrical data flow rate sequence, wherein adjacent electrical data flow rates are adjacent to each other and lag behind the electrical data flow rate in the electrical data flow rate sequence. The electrical data flow rate difference is calculated based on the electrical data flow rate and adjacent electrical data flow rates. The electrical data flow rate difference is the value obtained by subtracting the adjacent electrical data flow rates from the electrical data flow rate. By summing the differences in electrical data flow rates, a differential flow rate sequence is obtained.

5. The data migration method for multiple data sources as described in claim 4, characterized in that, The step of obtaining the optimal prediction model based on the stationary data flow rate sequence and the difference order includes: The candidate sets of autoregressive order and the candidate sets of shifted average order are identified. Autoregressive order candidate values ​​are extracted sequentially from the autoregressive order candidate set. Based on the extracted autoregressive order candidate values ​​and the shifted average order candidate set, an order combination group is obtained. The order combination group includes multiple order combinations, and each order combination includes one autoregressive order candidate value and one shifted average order candidate value. Summarize the order combination groups to obtain the order combination set. Extract order combinations from the order combination set one by one, and perform the following operations on each extracted order combination: The ARIMA model is constructed based on the difference order and the combination of orders. The ARIMA model is trained using a stationary data flow sequence to obtain the maximum likelihood function value. The total number of samples is obtained from the stationary data flow sequence, and the model BIC value is calculated based on the total number of samples, the maximum likelihood function value, and the difference order. The model BIC values ​​are summarized to obtain the model BIC value set. The order combination corresponding to the model with the smallest BIC value in the model BIC value set is taken as the optimal combination. The optimal prediction model is determined based on the optimal combination.

6. The data migration method for multiple data sources as described in claim 5, characterized in that, The formula for calculating the BIC value of the model is as follows: ， in, This represents the BIC value of the model. Indicates the candidate value of the autoregression order. Indicates the candidate values ​​for the order of the translational mean. Represents the total number of samples. Represents the logarithmic function. This represents the maximum likelihood function value.

7. The data migration method for multiple data sources as described in claim 6, characterized in that, The step of using a pre-built resource determination module to determine the predicted resource occupancy parameter set and obtain either a migrating node or a normal node includes: Based on the resource determination module, the lower and upper limits of resources are determined, and it is determined whether the predicted resource occupancy parameters in the predicted resource occupancy parameter set are all less than the lower limit of resources. If all predicted resource occupancy parameters in the predicted resource occupancy parameter set are less than the lower limit of resources, then the electrical data stream corresponding to the extracted electrical data stream data packet will be regarded as a resource surplus node. Otherwise, determine whether there is a predicted resource usage parameter in the set of predicted resource usage parameters that is greater than the resource limit. If there is a predicted resource occupancy parameter in the predicted resource occupancy parameter set that is greater than the resource limit, then the electrical data stream is regarded as a resource overload node. If there is no predicted resource usage parameter greater than the resource limit in the predicted resource usage parameter set, then the electrical data stream is treated as a normal node. Nodes with excess or overloaded resources will be designated as migration nodes.

8. The data migration method for multiple data sources as described in claim 7, characterized in that, The step of obtaining the optimal receiving node based on the migration node and the candidate receiving node set includes: If the node to be migrated is a node with surplus resources, then obtain the total amount of data to be migrated from the node with surplus resources, identify the set of surplus candidate response nodes from the set of candidate response nodes based on the node with surplus resources, and identify the best surplus response node from the set of surplus candidate response nodes based on the total amount of data to be migrated. If the node to be migrated is a resource overload node, then the node's predicted resource occupancy parameter set is determined based on the resource overload node, the maximum resource occupancy rate is determined based on the node's predicted resource occupancy parameter set, and the load reduction amount is calculated based on the maximum resource occupancy rate and the preset target resource occupancy rate. The load reduction amount is the value obtained by subtracting the target resource occupancy rate from the maximum resource occupancy rate. The required data flow rate is calculated based on the load reduction and maximum resource utilization. Overload migration data is obtained from the resource overload nodes based on the required data flow rate. An overload candidate response node set is obtained from the candidate response node set based on the resource overload nodes. The optimal overload response node is determined from the overload candidate response node set based on the overload migration data. The optimal excess response node or the optimal overload response node is taken as the optimal response node.

9. The data migration method for multiple data sources as described in claim 8, characterized in that, The step of identifying the optimal surplus response node from the surplus candidate response node set based on the total amount of data to be migrated includes: Extract excess candidate response nodes sequentially from the excess candidate response node set, obtain the excess candidate node resource occupancy rate set based on the extracted excess candidate response nodes, and predict the response occupancy rate set based on the total amount of data to be migrated and the excess candidate node resource occupancy rate set; If there is a response occupancy rate greater than the preset resource limit in the response occupancy rate set, the excess candidate response nodes are removed from the excess candidate response node set to obtain an updated excess candidate response node set. The updated excess candidate response node set is used as the excess candidate response node set, and the step of extracting excess candidate response nodes from the excess candidate response node set is returned until all excess candidate response nodes in the excess candidate response node set have been extracted. If there is no response occupancy rate greater than the resource limit in the response occupancy rate set, then the excess candidate response nodes are regarded as valid response nodes; Summarize the effective support nodes to obtain the effective support node set, and obtain the best surplus support node based on the effective support node set.

10. A data migration system with multiple data sources, characterized in that, The system includes: The data source connection module is used to identify the stream processing platform and multiple electrical data streams. The stream processing platform includes: multiple data source connectors, multiple connected data source connectors based on the multiple data source connectors and multiple electrical data streams, and electrical data stream data packet sets based on the multiple connected data source connectors. The node identification module is used to obtain the normal node set and the migration node set based on the electrical data stream data set, obtain the node topology diagram, identify the node topology diagram using the normal node set and the migration node set to obtain the identified topology diagram, extract the migration nodes from the migration node set in sequence, identify the target migration location in the identified topology diagram based on the extracted migration nodes, and obtain the candidate receiving node set from the identified topology diagram based on the target migration location. The optimal receiving node confirmation module is used to obtain the optimal receiving node based on the migration node and the candidate receiving node set, receive data migration instructions, and perform data migration on the migration node according to the optimal receiving node and the data migration instructions to obtain the migrated node. The electrical data migration module is used to summarize the migrated nodes, obtain the migrated node set, and complete the data migration method for multiple data sources based on the migrated node set.

Images