Data migration method and device, electronic equipment and storage medium

Abstract

The invention provides a data migration method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the index data of each node in real time, wherein the index data comprises a plurality of dimensions; determining a target dimension from the plurality of dimensions according to a data migration demand; determining the load of each node according to the target dimension; determining a migration strategy of the plurality of nodes according to the load, wherein the migration strategy is used for representing a corresponding relation between a data migration-out node and a data migration-in node in the plurality of nodes; generating a migration task according to the migration strategy; and executing the migration task to obtain a data migration result of the plurality of nodes. The invention relates to the technical field of independent stations, and can improve the data migration efficiency of an independent station server cluster.

Description

Technical Field

[0001] The present application relates to the technical field of independent stations, and in particular to a data migration method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of the Internet, more and more organizations tend to use Internet platforms to receive requests sent by users and provide corresponding services to users. For example, with the development of e-commerce, the emerging business model of cross-border e-commerce has also developed rapidly in recent years. Independent sites are a new form of cross-border e-commerce. In operation, they have high autonomy and are more flexible, avoiding many constraints of platform rules. Compared with traditional e-commerce platforms, the types of merchants running online stores in independent sites are different; the volume of users browsing online stores for online transactions is also different. In e-commerce platforms, messages published by back-end merchants are usually received through the message middleware of independent sites, and message subscription services are provided to users. Users usually send data operation requests to the back-end of independent sites through clients to obtain data services provided by e-commerce platforms. Due to the large number of Internet users in independent sites, the message middleware of independent sites usually receives a large number of messages from different sources and different contents in a unit time, resulting in reduced stability of message transmission in independent sites, which in turn leads to low efficiency of organizations in providing data services to users. Summary of the invention

[0003] In view of the above, it is necessary to propose a data migration method, device, electronic device and storage medium to solve the technical problem of low efficiency in data migration of message middleware of independent stations.

[0004] The present application provides a data migration method, which is applied to an electronic device, wherein the electronic device is communicatively connected to multiple nodes of a server, and the method includes: acquiring indicator data of each node in real time, wherein the indicator data includes multiple dimensions; determining a target dimension from the multiple dimensions according to data migration requirements; determining a load of each node according to the target dimension; determining a migration strategy for the multiple nodes according to the load, wherein the migration strategy is used to characterize the correspondence between data-migrating nodes and data-migrating nodes among the multiple nodes; generating a migration task according to the migration strategy; and executing the migration task to obtain data migration results for the multiple nodes.

[0005] In some embodiments, determining the migration strategy of the multiple nodes based on the load includes: determining the balance value of the target dimension in the indicator data of each node based on the load; determining a data migration-in node and a data migration-out node from the multiple nodes based on the balance value; and determining a migration strategy based on the data migration-out node and the data migration-in node.

[0006] In some embodiments, determining the migration strategy of the multiple nodes according to the load includes: determining the balance value of the target dimension in the indicator data of each node according to the load; determining the data migration-in node and the data migration-out node from the multiple nodes according to the balance value; determining the migration strategy according to the data migration-out node and the data migration-in node; predicting the value of each dimension in each node after executing the migration strategy according to the migration strategy; and determining the migration strategy when the values ​​of the multiple dimensions corresponding to each node are within a preset range.

[0007] In some embodiments, the method further includes: when the value of any one of the multiple dimensions corresponding to any node is not within the preset range, continuously updating the correspondence between the data migration-out nodes and the data migration-in nodes among the multiple nodes until the values ​​of the multiple dimensions corresponding to each node are within the preset range.

[0008] In some embodiments, determining the load of each node according to the target dimension includes: sorting the multiple nodes in order from small to large according to the target dimension; determining the order of each node and the corresponding value of the target dimension as the load of the node.

[0009] In some embodiments, executing the migration task to obtain the data migration results of the multiple nodes includes:

[0010] In the case where there are multiple migration tasks, the multiple migration tasks are divided into multiple batches according to a preset parallel number; wherein the number of migration tasks in each batch is less than or equal to the preset parallel number; the migration tasks in each batch are executed in parallel; after executing the migration tasks in all the batches, the data migration results are obtained.

[0011] In some embodiments, the multiple nodes are divided into multiple groups according to the label of each node, wherein each group of nodes has the same label; for any group of nodes, the balance value of the target dimension of each node in the any group of nodes is determined according to the load; a data migration node and a data migration node are determined from the any group of nodes according to the balance value; a migration strategy of the any group of nodes is determined according to the data migration node and the data migration node; and the migration strategy of each group of nodes is determined to obtain the migration strategy of the multiple nodes.

[0012] An embodiment of the present application also provides a data migration device, which includes: an acquisition module, used to acquire indicator data of each node in real time, and the indicator data includes multiple dimensions; a determination module, used to determine the target dimension from the multiple dimensions according to the data migration requirements; the determination module is also used to determine the load of each node according to the target dimension; the determination module is also used to determine the migration strategy of multiple nodes according to the load, and the migration strategy is used to characterize the correspondence between data migration-out nodes and data migration-in nodes among the multiple nodes; the determination module is also used to generate a migration task according to the migration strategy; the migration module is used to execute the migration task and obtain the data migration results of the multiple nodes.

[0013] An embodiment of the present application also provides an electronic device, which includes: a memory storing at least one instruction; and a processor executing the instruction stored in the memory to implement the data migration method.

[0014] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data migration method is implemented.

[0015] It can be seen from the above technical solutions that the embodiment of the present application detects the load imbalance in multiple nodes of the server by acquiring the indicator data of multiple nodes in the server in real time, and automatically performs load balancing operations without affecting the operation of the server to avoid configuration errors caused by manual intervention in the data migration process. The data migration strategy of multiple nodes is determined based on the indicator data of the server nodes, so that the migration strategy can be automatically adjusted according to the operating status of the server cluster, thereby improving the efficiency of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is an application scenario diagram of a data migration method provided in one embodiment of the present application.

[0017] Figure 2 This is a flow chart of a data migration method provided in one embodiment of the present application.

[0018] Figure 3 This is a flowchart of a method for determining the load of each node according to a target dimension provided by an embodiment of the present application.

[0019] Figure 4 This is a flowchart of a method for determining a migration strategy provided by an embodiment of the present application.

[0020] Figure 5 This is a flowchart of a method for determining a migration strategy provided by another embodiment of the present application.

[0021] Figure 6 It is a flowchart of a method for grouping multiple nodes and determining a migration strategy provided by an embodiment of the present application.

[0022] Figure 7 This is a flowchart of a method for executing a migration task according to a preset parallel quantity provided by an embodiment of the present application.

[0023] Figure 8 It is a functional module diagram of a data migration device provided in one embodiment of the present application.

[0024] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to more clearly understand the purpose, features and advantages of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other. In the following description, many specific details are set forth to facilitate a full understanding of the present application, and the embodiments described are only a part of the embodiments of the present application, rather than all of the embodiments.

[0026] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0028] An embodiment of the present application provides a data migration method, which can be applied to one or more electronic devices. The electronic device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0029] An electronic device can be any electronic product that can interact with a customer, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0030] The electronic device may also include a network device and / or a client device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud consisting of a large number of hosts or network servers based on cloud computing.

[0031] The network where the electronic device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0032] like Figure 1 The figure shows an application scenario diagram of a data migration method provided by an embodiment of the present application. The data migration method provided by the present application can be applied to an electronic device 100. The electronic device 100 is communicatively connected to an independent site 200, and is used to perform a migration operation on the data in the independent site 200.

[0033] The independent site 200 provided in this application is used to provide a trading platform for merchants and users of cross-border e-commerce. Among them, the independent site 200 includes a client 210, a message middleware 220, and a backend 230. Specifically, the client 210 is used to receive the request data input by the user into the independent site 200, the message middleware 220 is used to determine the business data to be searched according to the request information in the request data, and the backend 230 is used to publish business messages (for example, commodity information) in the message middleware 220, and the message middleware 220 determines the corresponding business message according to the request information sent by the client 210, and feeds the business message back to the user through the client 210. Among them, the message middleware 220 is a server for providing computing and storage resources, receiving and storing data, to support the operation of the independent site 200. The node can be a plurality of nodes corresponding to the server running the message middleware, wherein each node is used to manage the data stored in different topics and different partitions in the server.

[0034] like Figure 2 , which is a flow chart of a data migration method provided by an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. A data migration method provided by an embodiment of the present application includes the following steps.

[0035] S20, acquiring indicator data of each node in real time, wherein the indicator data includes multiple dimensions.

[0036] In one embodiment of the present application, the nodes may be multiple nodes corresponding to the server running the message middleware of the independent station, wherein each node is used to manage data stored in different topics and different partitions in the server. During the operation of the message middleware, in order to improve the efficiency of users obtaining independent station data through the message middleware, and to improve the timeliness of messages released to users by the back-end operation and maintenance personnel of the independent station, thereby improving the efficiency of data interaction, the indicator data of each node in the server during operation can be obtained in real time. Among them, the indicator data can be used to characterize the performance of the server during the working process.

[0037] Among them, the message middleware can receive, store and forward messages, and manage the Broker agent node of the message queue. For the monitoring and evaluation of Broker, a series of indicator data are usually focused on to ensure the stability and performance of the system.

[0038] Specifically, the indicator data of the server running the message middleware has multiple dimensions, each dimension is used to describe the performance status of each node in the server from a different perspective. Exemplarily, the multiple dimensions in the indicator data include: basic performance indicators, resource usage, queue and partition status, system-specific indicators, and cluster status indicators.

[0039] Among them, the basic performance indicators include server throughput, which is used to measure the speed at which a node processes messages. The server throughput can be measured by the number of messages processed per second (Messages Per Second, MPS) or the number of bytes processed per second (Bytes Per Second, BPS). The basic performance indicators also include node latency, which is used to indicate the time interval from when a message is sent to when it is processed. This indicator can be used to characterize the response speed of a node. The basic performance indicators also include node availability. Specifically, node availability is used to indicate the proportion of time that a node can provide services normally, usually expressed as a percentage. Among them, resource usage includes central processing unit CPU usage, which is used to measure the proportion of CPU resources occupied by any node. When the CPU usage ratio is high, it indicates that the performance of the node is insufficient to process the current message data, so the node may be under a high load condition. Resource usage also includes memory usage, which is used to characterize the proportion of memory resources occupied by a node. When the memory usage ratio is high, the efficiency of the node in processing the current message may be unstable or there may be performance degradation. Resource usage also includes disk read and write rates and disk space utilization. When the disk read and write rates are high, the amount of data currently read from or written to the disk may be high, indicating that the disk load is high, which may cause disk performance degradation and, in turn, server performance degradation.

[0040] In one embodiment of the present application, in order to avoid the problem of excessive computing resource usage of the server due to monitoring the indicator data of each node, the indicator data of each node can be periodically obtained, so as to avoid excessive computing resource usage caused by the real-time operation of the monitoring program, resulting in a decrease in server performance. Exemplarily, the indicator data of the server can be obtained at intervals of a preset period. Exemplarily, the preset period can be 30 seconds, 1 minute, or 2 minutes, and the present application does not limit this.

[0041] S21: Determine a target dimension from the multiple dimensions according to data migration requirements.

[0042] In one embodiment of the present application, the data migration requirement is used to characterize the state that the index data of each node in the server needs to reach after the data migration is performed on each node. Therefore, in the process of performing multi-node data migration to achieve server load balancing, one or more target dimensions can be determined from the index data of multiple dimensions. The above multiple dimensions can be used to characterize the feasibility and efficiency of data migration, and can also characterize the stability and performance of the server after data migration.

[0043] In an embodiment of the present application, when the data migration requirement indicates that the performance and capacity of each node in the server must be ensured to reach a load-balanced state, the throughput of each node can be determined as the target dimension, wherein the throughput is used to characterize the ability of each node in the current server to process messages per unit time. After data migration, the throughput corresponding to each node of the server should reach a balanced state.

[0044] When the data migration requirement indicates that the message processing delay of each node in the server needs to be lower than a certain threshold, the message delay of each node can be determined as the target dimension. In this way, after the data migration, the time for sending, receiving and processing messages of each node can be ensured to be relatively balanced, which can meet the real-time requirements of each node for the data migration requirement.

[0045] When the data migration requirements indicate that it is necessary to ensure that the storage capacity of each node in the server is balanced, the storage capacity of each node can be determined as the target dimension. In this way, after the data migration, the difference between the storage capacity of each node can be ensured to be small, thereby optimizing the server's disk space, ensuring that each node has sufficient storage capacity to store messages, and improving the stability of the server.

[0046] In one embodiment of the present application, different data migration requirements correspond to different node performance requirements. When the server performance is relatively strong, a target dimension can be determined from multiple dimensions based on the data migration requirements, thereby reducing the pressure on the server caused by data migration while achieving multi-node load balancing. When the server performance is relatively weak, multiple target dimensions can be determined from multiple dimensions based on the data migration requirements, thereby constraining the performance of the nodes based on more dimensions while achieving multi-node load balancing, thereby improving the degree of balance of each node in multiple dimensions.

[0047] S22: Determine the load of the server according to the target dimension.

[0048] In one embodiment of the present application, in order to ensure that the load of each node can be balanced after the data of each node in the server is migrated, the load condition of each node can be determined based on the indicator data to determine whether there is an imbalance.

[0049] In one embodiment of the present application, the index data of the middleware server can be used to determine the load of each node, so that the operating status of the server can be determined according to the index data obtained in real time, and then the problems that occur during the operation of the server can be determined in time. In order to avoid the excessive amount of index data used in the process of determining the load of the server, the time range of the index data used to evaluate the load of the server can be predefined, so that the index data can be periodically collected from a period of time in the past according to a preset time window. For example, when the preset time window is 10 minutes and the preset period is 1 minute, the index data of each server is collected every 1 minute in the past 10 minutes. This can improve the accuracy of the prediction, rich data samples: by periodically collecting data, rich historical data samples can be accumulated. These data contain the performance of the server under different loads and different time periods, providing a more comprehensive input for the prediction model, which helps to improve the accuracy and reliability of the prediction. Capture periodic changes: The load of the server often shows certain periodic changes, such as the difference between working days and rest days, the difference between day and night, etc. Periodic data collection can better capture these periodic changes, so that the prediction model can more accurately reflect the actual load of the server. It can also support more complex prediction models, time series analysis: periodically collected data can form a time series, which supports the use of time series analysis models for prediction. The time series analysis model can capture the time dependency in the data, so as to more accurately predict future load trends. Machine learning algorithm: For complex server load prediction problems, machine learning algorithms can be used to build prediction models. The periodically collected data provides sufficient training samples for the machine learning algorithm, which helps the algorithm learn the laws and patterns of server load changes. It can also monitor and adjust in real time to detect abnormalities in time: by periodically collecting data, the performance indicators of the server can be monitored in real time, and abnormal situations can be detected and handled in time. This helps to avoid server failures due to overload and ensure the stability and reliability of the service. Dynamic adjustment strategy: based on the periodically collected data, the server's load distribution strategy and resource scheduling strategy can be dynamically adjusted. When it is predicted that the server load will exceed its carrying capacity, resource expansion or load balancing adjustment can be carried out in advance to cope with the upcoming peak load. It can also optimize resource utilization and allocate resources reasonably: the periodically collected data can help administrators understand the resource usage of the server, including CPU usage, memory usage, disk I / O, etc. Based on this data, resources can be allocated reasonably to avoid resource waste and bottleneck problems. Reduce costs: By optimizing resource utilization, the operating costs of the server can be reduced. For example, during periods of low load, the number of running servers can be reduced or their performance configuration can be lowered to save energy and reduce operation and maintenance costs.It can also support decision-making by providing data support: The periodically collected data provides administrators with rich data support, helping them to more accurately understand the performance and load of the server. These data can be used as a basis for decision-making, helping administrators make more reasonable decisions. Evaluate system performance: By comparing data collected in different time periods, the performance changes and optimization effects of the system can be evaluated. This helps administrators understand the stability and reliability of the system, and whether there are areas that need improvement.

[0050] In one embodiment of the present application, when evaluating the load of each node according to the target dimension in the indicator data obtained in real time, the evaluation can be performed according to the following dimensions. For example, the load of the node can be evaluated according to the computing resource occupancy rate or memory usage rate of the node. When the computing resource occupancy rate or memory usage rate of the node is high, it indicates that the load of the node is too high. For another example, the load of the node can be evaluated according to the disk occupancy rate or disk response time of the node. When the disk occupancy rate is high, it indicates that the amount of data stored in the node is too high, and the load of the node is high; when the disk response time is long, it indicates that the time taken for the node to execute data operation instructions is high, and the load of the node is high. For another example, the load of the node can be evaluated according to the request response time or throughput of the node. When the response time of the node is long, it indicates that the node takes a long time to process messages, and the load of the node is high; when the throughput of the node is too high, it indicates that the number of messages currently distributed by the node is high, and the load of the node is high. For another example, the node load can also be evaluated based on the number of concurrent connections of the node or the length of the message queue in the node. When the number of concurrent connections of the node is too high, it indicates that the server has assigned too many data processing tasks to the node, indicating that the node load is too high; when the length of the node message queue is long, it indicates that the number of messages to be processed in the node is too large, indicating that the node load is high. For more details, please refer to the method of determining the load of each node according to the target dimension. Figure 3 The corresponding detailed description.

[0051] S23: Determine migration strategies for the multiple nodes according to the loads.

[0052] In an embodiment of the present application, when the loads corresponding to the nodes in the server are different, the migration strategies corresponding to the multiple nodes in the server can be determined according to the loads corresponding to each node. The migration strategy is used to characterize the corresponding relationship between the data migration-out nodes and the data migration-in nodes in the multiple nodes, and the data in the data migration-out nodes in the server can be subsequently migrated to the data migration-in nodes according to the migration strategy.

[0053] In one embodiment of the present application, the order of the loads of all nodes from small to large can be determined based on the load of each node, and the average level corresponding to the loads of multiple nodes can be determined, and then the data migration nodes with loads higher than the average level and the data migration nodes with loads lower than the average level can be determined among the multiple nodes, and the migration strategy can be determined based on the migration nodes and the migration nodes. Among them, the data in the data migration node needs to be migrated out to the data migration node. For details, please refer to the specific method of determining the migration strategy. Figure 4 The corresponding detailed description.

[0054] In another embodiment of the present application, in order to ensure that after data migration is performed on the nodes in the server according to the migration strategy, the data of multiple dimensions in the indicator data of each node are close to a load-balanced state, after determining the data migration-out node and the data migration-in node according to the load of each node, the value of each dimension in each node after executing the migration strategy is predicted, and the migration strategy is determined based on the value of each dimension. For example, when the value of each dimension of each node is not within a preset range, it is determined that the migration strategy needs to be optimized. When it is determined that the migration strategy needs to be optimized based on the value of each dimension, the correspondence between the data migration-out node and the data migration-in node can be updated until the value of each dimension of each node is within the preset range, and the update is stopped and the migration strategy is obtained. For specific methods of determining the migration strategy, please refer to Figure 5 The corresponding detailed description.

[0055] In another embodiment of the present application, multiple nodes can also be sorted according to multiple target dimensions (such as node load, response time, and throughput, etc.) at the same time. Specifically, the weight of each dimension can be pre-set to characterize the importance of each dimension in the load balancing process. Each dimension is then standardized to obtain a standardized score for each dimension, and multiple nodes are sorted based on the weight of each dimension and the standardized score.

[0056] In an embodiment of the present application, the nodes may be grouped according to the node labels, and the migration strategy of each group of nodes may be determined according to the load of each group of nodes. Figure 6 The corresponding detailed description.

[0057] S24: Generate a migration task according to the migration strategy.

[0058] In one embodiment of the present application, the data type, data volume, and data migration node to be migrated can be determined according to the migration strategy. And the usage scenario, performance requirements, and security requirements of the migrated data can be determined. This ensures that the data migration requirements can be achieved after the migration strategy is executed, such as improving data access speed, reducing storage costs, or enhancing data security. Among them, the migration task can be an instruction with data migration function, and when the instruction is executed, the migration strategy corresponding to the data migration requirement can be implemented.

[0059] S25, executing the migration task to obtain data migration results of the multiple nodes.

[0060] In one embodiment of the present application, during the execution of the migration task, the migration progress and the performance of each node in the server can be monitored in real time to ensure that the migration process can proceed smoothly. In order to improve the execution efficiency of the migration task, multiple migration tasks can be processed in parallel. However, the process of processing multiple migration tasks in parallel may occupy more computing resources of the server. In order to ensure stable operation of the server, the number of parallel tasks can be pre-set, and the migration tasks can be executed in batches according to the number of parallel tasks. For specific methods of executing migration tasks according to the preset number of parallel tasks, please refer to Figure 7 The corresponding detailed description.

[0061] It can be seen from the above technical solutions that the embodiment of the present application detects the load imbalance in multiple nodes of the server by acquiring the indicator data of multiple nodes in the server in real time, and automatically performs load balancing operations without affecting the operation of the server to avoid configuration errors caused by manual intervention in the data migration process. The data migration strategy of multiple nodes is determined based on the indicator data of the server nodes, so that the migration strategy can be automatically adjusted according to the operating status of the server cluster, thereby improving the efficiency of data migration.

[0062] like Figure 3 , is a flow chart of a method for determining the load of each node provided in an embodiment of the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for determining the load of each node provided in an embodiment of the present application includes the following steps.

[0063] S30, sorting the multiple nodes in ascending order according to the target dimension to obtain the order of each node.

[0064] In one embodiment of the present application, in order to determine the load of each node, multiple nodes may be sorted in order from small to large according to the target dimension to obtain the order of each node. For example, if the target dimension is throughput, and the throughput of node A is 10 messages per minute, the throughput of node B is 5 messages per minute, the throughput of node C is 15 messages per minute, the throughput of node D is 20 messages per minute, and the throughput of node E is 25 messages per minute. Then, the multiple nodes are sorted in order from small to large according to the throughput: node B, node A, node C, node D and node E.

[0065] S31, determining the order of each node and the corresponding target dimension value as the load of each node.

[0066] In one embodiment of the present application, the load of each node can be characterized according to the order of each node and the value of the corresponding target dimension. Since multiple nodes are arranged in ascending order according to the target dimension, the higher the order of the node, the smaller the load of the node and the smaller the value of the corresponding target dimension.

[0067] like Figure 4 FIG. 1 is a flowchart of a method for determining a migration strategy provided in an embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for determining a migration strategy provided in an embodiment of the present application includes the following steps.

[0068] S40: Determine a balance value of the target dimension in the indicator data of each node according to the load.

[0069] In one embodiment of the present application, in order to determine the migration strategies corresponding to multiple nodes in the server according to the load corresponding to each node, the balance value of the target dimension of each node can be determined according to the load. The balance value can be the median of the target dimension of each node, or the mean of the target dimension of each node, which is not limited in the present application.

[0070] S41, determining a data migration node and a data migration node from the multiple nodes according to the balance value.

[0071] In one embodiment of the present application, a node whose target dimension value is greater than the equilibrium value may be determined as a data migration node, and a node whose target dimension is less than or equal to the equilibrium value may be determined as a data migration node.

[0072] S42: Determine a migration strategy according to the data migration-out node and the data migration-in node.

[0073] In one embodiment of the present application, the correspondence between the data-migrating nodes and the data-migrating nodes can be determined according to the value of the target dimension of each data-migrating node and the value of the target dimension of each data-migrating node, thereby determining the migration strategy.

[0074] like Figure 5 FIG. 1 is a flowchart of a method for determining a migration strategy provided by another embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for determining a migration strategy provided by an embodiment of the present application includes the following steps.

[0075] S50: Determine a balance value of the target dimension in the indicator data of each node according to the load.

[0076] In one embodiment of the present application, in order to determine the migration strategies corresponding to multiple nodes in the server according to the load corresponding to each node, the balance value of the target dimension of each node can be determined according to the load. The balance value can be the median of the target dimension of each node, or the mean of the target dimension of each node, which is not limited in the present application.

[0077] S51, determining a data migration node and a data migration node from the multiple nodes according to the balance value.

[0078] In one embodiment of the present application, a node whose target dimension value is greater than the equilibrium value may be determined as a data migration node, and a node whose target dimension is less than or equal to the equilibrium value may be determined as a data migration node.

[0079] S52: Determine a migration strategy according to the data migration-out node and the data migration-in node.

[0080] In one embodiment of the present application, the correspondence between the data-migrating nodes and the data-migrating nodes can be determined according to the value of the target dimension of each data-migrating node and the value of the target dimension of each data-migrating node, thereby determining the migration strategy.

[0081] S53: predicting, according to the migration strategy, the value of each dimension in each node after executing the migration strategy.

[0082] In one embodiment of the present application, in order to determine whether the migration strategy needs to be optimized, the value of each dimension in each node after the execution of the migration strategy can be simulated, so that the operating status of multiple nodes of the server after the execution of the migration strategy can be determined based on the value of each simulated dimension before executing the migration strategy.

[0083] S54, when the values ​​of the multiple dimensions corresponding to each node are within a preset range, determine to obtain the migration strategy.

[0084] In one embodiment of the present application, when the values ​​of multiple dimensions corresponding to any node are within a preset range, it indicates that the indicator data of the node are close to the equilibrium value. Therefore, when the values ​​of multiple dimensions corresponding to each node are within the preset range, the migration data of multiple nodes can be determined.

[0085] In one embodiment of the present application, the method also includes: when the value of any one of the multiple dimensions corresponding to any node is not within the preset range, continuously updating the correspondence between the data migration-out nodes and the data migration-in nodes among the multiple nodes until the values ​​of the multiple dimensions corresponding to each node are within the preset range.

[0086] like Figure 6 , is a flowchart of a method for grouping multiple nodes and determining a migration strategy provided by another embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for grouping multiple nodes and determining a migration strategy provided by an embodiment of the present application includes the following steps.

[0087] S60, dividing the multiple nodes into multiple groups according to the label of each node, wherein each group of nodes has the same label.

[0088] In one embodiment of the present application, the label of the node is used to characterize the number of the server device to which the node belongs. When the labels of any two nodes are the same, it indicates that the two nodes belong to the same server device. In order to improve the efficiency of data migration, multiple nodes can also be grouped according to the labels of the nodes to ensure that each group of nodes has the same label, so that data migration can be ensured within the server device to avoid cross-device situations, thereby improving the efficiency of data migration.

[0089] S61: For any group of nodes, determine the balancing value of the target dimension of each node in the any group of nodes according to the load.

[0090] In an embodiment of the present application, in order to determine the migration strategies corresponding to multiple nodes in the server according to the load corresponding to each node, the balance value of the target dimension of each node in any group of nodes can be determined according to the load. The balance value can be the median of the target dimension of each node, or the mean of the target dimension of each node, which is not limited in the present application.

[0091] S62: Determine a data migration node and a data migration node from the arbitrary group of nodes according to the balance value.

[0092] In one embodiment of the present application, in any group of nodes, nodes whose target dimension values ​​are greater than the equilibrium value can be determined as data migration nodes, and nodes whose target dimension values ​​are less than or equal to the equilibrium value can be determined as data migration nodes.

[0093] S63: Determine a migration strategy of the arbitrary group of nodes according to the data-migrating node and the data-migrating node.

[0094] In one embodiment of the present application, for any group of nodes, the correspondence between the data-migrating nodes and the data-migrating nodes can be determined according to the value of the target dimension of each data-migrating node and the value of the target dimension of each data-migrating node, thereby determining the migration strategy.

[0095] S64, determining a migration strategy for each group of nodes to obtain the migration strategies of the multiple nodes.

[0096] like Figure 7 , is a flowchart of a method for executing a migration task according to a preset parallel number provided by another embodiment of the present application. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. The method for executing a migration task according to a preset parallel number provided by an embodiment of the present application includes the following steps.

[0097] S70, when there are multiple migration tasks, divide the multiple migration tasks into multiple batches according to a preset parallel number; wherein the number of migration tasks in each batch is less than or equal to the preset parallel number.

[0098] In one embodiment of the present application, during the execution of the migration task, the migration progress and the performance of each node in the server can be monitored in real time to ensure that the migration process can proceed smoothly. In order to improve the execution efficiency of the migration task, multiple migration tasks can be processed in parallel. However, the process of processing multiple migration tasks in parallel may occupy more computing resources of the server. In order to ensure stable operation of the server, the number of parallel tasks can be pre-set, and the migration tasks can be executed in batches according to the number of parallel tasks. Among them, the number of migration tasks in each batch is less than or equal to the preset number of parallel tasks.

[0099] S71, executing the migration tasks of each batch in parallel.

[0100] In an embodiment of the present application, for any batch of migration tasks, the migration tasks in the batch can be executed in parallel. Specifically, the migration tasks in each batch can be executed in a multi-threaded or multi-concurrent manner, thereby improving the efficiency of executing the migration tasks.

[0101] S72: After executing the migration tasks of all the batches, obtain the data migration results.

[0102] See also Figure 8 , Figure 8 1 is a functional module diagram of a data migration device provided in an embodiment of the present application. The data migration device 91 includes an acquisition module 910, a determination module 920 and a migration module 930. The module / unit referred to in the present application refers to a series of computer-readable instruction segments that can be executed by the processor 13 and can complete fixed functions, which are stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0103] The acquisition module 910 is used to acquire the indicator data of each node in real time, and the indicator data includes multiple dimensions.

[0104] The determination module 920 is used to determine a target dimension from the multiple dimensions according to data migration requirements.

[0105] The determination module 920 is further configured to determine the load of each node according to the target dimension.

[0106] The determination module 920 is further configured to determine a migration strategy for a plurality of nodes according to the load, wherein the migration strategy is configured to characterize a correspondence between data-migrating nodes and data-migrating nodes in the plurality of nodes.

[0107] The determination module 920 is further configured to generate a migration task according to the migration strategy.

[0108] The migration module 930 is used to execute the migration task and obtain the data migration results of the multiple nodes.

[0109] In some embodiments, the determination module 920 is also used to determine the balance value of the target dimension in the indicator data of each node according to the load; determine the data migration node and the data migration node from the multiple nodes according to the balance value; and determine the migration strategy based on the data migration node and the data migration node.

[0110] In some embodiments, the determination module 920 is also used to determine the balance value of the target dimension in the indicator data of each node according to the load; determine the data migration node and the data migration node from the multiple nodes according to the balance value; determine the migration strategy according to the data migration node and the data migration node; predict the value of each dimension in each node after executing the migration strategy according to the migration strategy; and determine the migration strategy when the values ​​of the multiple dimensions corresponding to each node are within a preset range.

[0111] In some embodiments, the determination module 920 is also used to continuously update the correspondence between the data migration nodes and the data migration nodes in the multiple nodes when the value of any one of the multiple dimensions corresponding to any node is not within the preset range, until the values ​​of the multiple dimensions corresponding to each node are within the preset range.

[0112] In some embodiments, the determination module 920 is further used to sort the multiple nodes in order from small to large according to the target dimension; determine the order of each node and the corresponding value of the target dimension as the load of the node.

[0113] In some embodiments, the migration module 930 is also used to divide the multiple migration tasks into multiple batches according to a preset parallel number when there are multiple migration tasks; wherein the number of migration tasks in each batch is less than or equal to the preset parallel number; execute the migration tasks of each batch in parallel; and obtain the data migration results after executing the migration tasks of all the batches.

[0114] In some embodiments, the determination module 920 is also used to divide the multiple nodes into multiple groups according to the label of each node, wherein each group of nodes has the same label; for any group of nodes, determine the balance value of the target dimension of each node in the any group of nodes according to the load; determine the data migration node and the data migration node from the any group of nodes according to the balance value; determine the migration strategy of the any group of nodes according to the data migration node and the data migration node; determine the migration strategy of each group of nodes to obtain the migration strategy of the multiple nodes.

[0115] See also Fig. 9 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 100 includes a memory 12 and a processor 13. The memory 12 is used to store computer-readable instructions, and the processor 13 is used to execute the computer-readable instructions stored in the memory to implement a data migration method described in any of the above embodiments.

[0116] In an embodiment of the present application, the electronic device 100 further includes a bus, and a computer program stored in the memory 12 and executable on the processor 13, such as a data migration program.

[0117] Fig. 9 Only the electronic device 100 having the memory 12 and the processor 13 is shown, and those skilled in the art can understand that Fig. 9The structure shown does not constitute a limitation on the electronic device 100 , and the electronic device 100 may include fewer or more components than shown in the figure, or combine some components, or arrange the components differently.

[0118] Combination Figure 2 The memory 12 in the electronic device 100 stores a plurality of computer-readable instructions to implement the data migration method, and the processor 13 can execute the plurality of instructions to implement: real-time acquisition of indicator data of each node, the indicator data including a plurality of dimensions; determining a target dimension from the plurality of dimensions according to data migration requirements; determining a load of each node according to the target dimension; determining a migration strategy of the plurality of nodes according to the load, the migration strategy being used to characterize a correspondence between data-migrating nodes and data-migrating nodes in the plurality of nodes; generating a migration task according to the migration strategy; and executing the migration task to obtain data migration results of the plurality of nodes.

[0119] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0120] Those skilled in the art will appreciate that the schematic diagram is merely an example of the electronic device 100 and does not constitute a limitation on the electronic device 100. The electronic device 100 may be a bus-type structure or a star-type structure. The electronic device 100 may also include more or less other hardware or software than shown in the diagram, or a different arrangement of components. For example, the electronic device 100 may also include input and output devices, network access devices, etc.

[0121] It should be noted that the electronic device 100 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0122] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium can be non-volatile or volatile. The readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 100, such as a mobile hard disk of the electronic device 100. In other embodiments, the memory 12 can also be an external storage device of the electronic device 100, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 100. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 100, such as a code of a data migration program, etc., but also can be used to temporarily store data that has been output or is to be output.

[0123] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 100, and uses various interfaces and lines to connect various components of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing programs or modules stored in the memory 12 (for example, executing data migration programs, etc.), and calling data stored in the memory 12.

[0124] The processor 13 executes the operating system of the electronic device 100 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned various data migration method embodiments, for example Figure 2 Steps shown.

[0125] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 100. For example, the computer program may be divided into an acquisition module 910, a determination module 920, and a migration module 930.

[0126] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute a part of a data migration method described in each embodiment of the present application.

[0127] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware devices to complete through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.

[0128] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory and other memory, etc.

[0129] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0130] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. Fig. 9 Only one arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. The bus is configured to realize the connection and communication between the memory 12 and at least one processor 13, etc.

[0131] An embodiment of the present application also provides a computer-readable storage medium (not shown), in which computer-readable storage medium is stored computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the data migration method described in any of the above embodiments.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0133] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0135] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the specification can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solution of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present application.

Claims

1. A data migration method, applied to an electronic device, wherein the electronic device is communicatively connected to a plurality of nodes of a server, characterized in that: The method comprises: Acquire the index data of each node in real time, wherein the index data includes multiple dimensions; Determining a target dimension from the multiple dimensions according to data migration requirements; Determine the load of each node according to the target dimension; Determine a migration strategy for the multiple nodes according to the load, wherein the migration strategy is used to characterize a corresponding relationship between data-migrating nodes and data-migrating nodes in the multiple nodes; Generate a migration task according to the migration strategy; The migration task is executed to obtain data migration results of the multiple nodes.

2. The data migration method according to claim 1, characterized in that: The step of determining the migration strategies of the plurality of nodes according to the load comprises: Determine a balance value of the target dimension in the indicator data of each node according to the load; Determine a data migration node and a data migration node from the plurality of nodes according to the balance value; A migration strategy is determined according to the data migration-out node and the data migration-in node.

3. The data migration method according to claim 1, characterized in that: The step of determining the migration strategies of the plurality of nodes according to the load comprises: Determine a balance value of the target dimension in the indicator data of each node according to the load; Determine a data migration node and a data migration node from the plurality of nodes according to the balance value; Determine a migration strategy according to the data migration-out node and the data migration-in node; Predicting, according to the migration strategy, the value of each dimension in each node after executing the migration strategy; When the values ​​of the multiple dimensions corresponding to each node are within a preset range, it is determined that the migration strategy is obtained.

4. The data migration method according to claim 3, characterized in that: The method further comprises: When the value of any one of the multiple dimensions corresponding to any node is not within the preset range, the correspondence between the data migration-out nodes and the data migration-in nodes in the multiple nodes is continuously updated until the values ​​of the multiple dimensions corresponding to each node are within the preset range.

5. The data migration method according to claim 1, characterized in that: The determining the load of each node according to the target dimension includes: Sorting the multiple nodes in ascending order according to the target dimension; Determine the order of each node and the corresponding value of the target dimension as the load of the node.

6. The data migration method according to claim 1, characterized in that: The performing of the migration task to obtain the data migration results of the multiple nodes includes: In the case where there are multiple migration tasks, the multiple migration tasks are divided into multiple batches according to a preset parallel number; wherein the number of migration tasks in each batch is less than or equal to the preset parallel number; Execute the migration tasks of each batch in parallel; After executing the migration tasks of all the batches, the data migration results are obtained.

7. The data migration method according to claim 1, characterized in that: The method further comprises: Dividing the plurality of nodes into a plurality of groups according to the label of each node, wherein each group of nodes has the same label; For any group of nodes, determining a balancing value of the target dimension of each node in the any group of nodes according to the load; Determine a data migration node and a data migration node from the arbitrary group of nodes according to the balance value; Determine a migration strategy of the arbitrary group of nodes according to the data migration-out node and the data migration-in node; A migration strategy for each group of nodes is determined to obtain the migration strategies for the multiple nodes.

8. A data migration device, characterized in that: The device comprises a module for implementing the data migration method according to any one of claims 1 to 7, and the device comprises: An acquisition module is used to acquire the indicator data of each node in real time, wherein the indicator data includes multiple dimensions; A determination module, configured to determine a target dimension from the multiple dimensions according to data migration requirements; The determination module is further used to determine the load of each node according to the target dimension; The determination module is further used to determine a migration strategy of multiple nodes according to the load, wherein the migration strategy is used to characterize a corresponding relationship between data-migrating nodes and data-migrating nodes in the multiple nodes; The determination module is further configured to generate a migration task according to the migration strategy; The migration module is used to execute the migration task and obtain the data migration results of the multiple nodes.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor is used to implement the data migration method as described in any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the data migration method according to any one of claims 1 to 7 are implemented.

Images