Data migration method, device, computer equipment and storage medium

Abstract

The present application relates to a data migration method, device, computer equipment, storage medium and computer program product. The method includes: when a main cluster fails, obtaining the attribute information of the backup cluster corresponding to the main cluster; exporting the historical data in the main cluster, transmitting the historical data to the backup cluster corresponding to the attribute information according to the attribute information, and obtaining the real-time data in the main cluster, transmitting the real-time data to the backup cluster; using the backup cluster as a new data source for the secondary cluster, taking over the service of the main cluster, and performing real-time data transmission to the secondary cluster. The use of this method can improve the stability of the data warehouse.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data migration method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of database clusters, the data capacity of data warehouses is getting larger and larger. With the explosive growth of data volume and the increasing requirements for data real-time performance, database nodes in the cluster may fail or be shut down for a long time, causing service interruptions and affecting the availability of data warehouses.

[0003] In traditional technology, when a data warehouse fails, that is, when the main database goes down or fails, in the main cluster, a standby node corresponding to the main node where the main database is located is determined, and the uncommitted transactions and the latest data updates in the main node are synchronized to the standby node, and the connection information between other nodes and the main node is updated to the information connected to the standby node.

[0004] However, in the current traditional technology, when all nodes of the main cluster are unavailable, the backup nodes are also unavailable, and the main node of the main cluster cannot be switched between the main and backup nodes, resulting in poor stability of the data warehouse. Summary of the invention

[0005] Based on this, it is necessary to provide a data migration method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0006] In a first aspect, the present application provides a data migration method, comprising:

[0007] When a main cluster fails, obtaining attribute information of a backup cluster corresponding to the main cluster;

[0008] Exporting historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the primary cluster, and transmitting the real-time data to the standby cluster;

[0009] The standby cluster is used as a new data source for the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster.

[0010] In one of the embodiments, when the primary cluster fails, before obtaining the attribute information of the backup cluster corresponding to the primary cluster, the method further includes:

[0011] Send heartbeat signals to the main cluster and obtain them in real time;

[0012] The delay information of the main cluster is determined based on the heartbeat signal of the main cluster. When the delay information is greater than a preset threshold, the delay information is displayed on a display page and an alarm message is sent.

[0013] In one embodiment, exporting the historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring the real-time data in the primary cluster, and transmitting the real-time data to the standby cluster includes:

[0014] Exporting the historical data in the primary cluster according to a preset data format, determining a standby cluster according to the identifier in the attribute information, and transmitting the exported historical data to the standby cluster according to the preset data format;

[0015] Real-time data is acquired, the real-time data is stored in a preset storage space, and the standby cluster is instructed to start a real-time program; the real-time program is used for the standby cluster to receive the real-time data in real time.

[0016] In one embodiment, the step of using the standby cluster as a new data source for the secondary cluster, taking over the service of the primary cluster, and performing real-time data transmission to the secondary cluster includes:

[0017] Updating the first connection information of the secondary cluster corresponding to the primary cluster to the second connection information corresponding to the standby cluster;

[0018] The standby cluster is instructed to synchronize data with the secondary cluster, and service traffic migration from the primary cluster to the standby cluster is completed.

[0019] In one embodiment, the method further comprises:

[0020] When the main cluster is restored to a normal operating state, the real-time data is synchronized back to the main cluster, and the main cluster is used as a data source to perform real-time data transmission to the secondary cluster.

[0021] In one of the embodiments, when the primary cluster fails, before obtaining the attribute information of the backup cluster corresponding to the primary cluster, the method further includes:

[0022] When the main cluster operates normally, backing up all data of the main cluster to the standby cluster according to a first preset period;

[0023] According to a second preset period, the incremental data of the primary cluster is backed up to the standby cluster.

[0024] In a second aspect, the present application further provides a data migration device, including:

[0025] An acquisition module, used to acquire attribute information of a standby cluster corresponding to the main cluster when a main cluster fails;

[0026] A first transmission module, configured to export the historical data in the primary cluster, transmit the historical data to a standby cluster corresponding to the attribute information according to the attribute information, obtain the real-time data in the primary cluster, and transmit the real-time data to the standby cluster;

[0027] The second transmission module is used to use the standby cluster as a new data source of the secondary cluster, take over the service of the primary cluster, and perform real-time data transmission to the secondary cluster.

[0028] In one embodiment, the device further comprises:

[0029] The sending module is used to send heartbeat signals to the main cluster and obtain them in real time;

[0030] The alarm module is used to determine the delay information of the main cluster based on the heartbeat signal of the main cluster, and when the delay information is greater than a preset threshold, the delay information is displayed on a display page and an alarm message is sent.

[0031] In one of the embodiments, the first transmission module is specifically used to export the historical data in the primary cluster according to a preset data format, determine the standby cluster according to the identifier in the attribute information, and transmit the exported historical data to the standby cluster according to the preset data format;

[0032] Real-time data is acquired, the real-time data is stored in a preset storage space, and the standby cluster is instructed to start a real-time program; the real-time program is used for the standby cluster to receive the real-time data in real time.

[0033] In one of the embodiments, the transmission module is specifically used to update the first connection information of the secondary cluster corresponding to the primary cluster to the second connection information corresponding to the standby cluster;

[0034] The standby cluster is instructed to synchronize data with the secondary cluster, and service traffic migration from the primary cluster to the standby cluster is completed.

[0035] In one embodiment, the device further comprises:

[0036] The third transmission module is used to synchronize the real-time data back to the main cluster when the main cluster recovers to a normal operating state, and use the main cluster as a data source to perform real-time data transmission to the secondary cluster.

[0037] In one embodiment, the device further comprises:

[0038] A first backup module, configured to back up the full amount of data of the primary cluster to the backup cluster according to a first preset period when the primary cluster operates normally;

[0039] The second backup module is used to back up the incremental data of the primary cluster to the backup cluster according to a second preset period.

[0040] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] When a main cluster fails, obtaining attribute information of a backup cluster corresponding to the main cluster;

[0042] Exporting historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the primary cluster, and transmitting the real-time data to the standby cluster;

[0043] The standby cluster is used as a new data source for the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster.

[0044] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0045] When a main cluster fails, obtaining attribute information of a backup cluster corresponding to the main cluster;

[0046] Exporting historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the primary cluster, and transmitting the real-time data to the standby cluster;

[0047] The standby cluster is used as a new data source for the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster.

[0048] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0049] When a main cluster fails, obtaining attribute information of a backup cluster corresponding to the main cluster;

[0050] Exporting historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the primary cluster, and transmitting the real-time data to the standby cluster;

[0051] The standby cluster is used as a new data source for the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster.

[0052] The above-mentioned data migration method, device, computer equipment, storage medium and computer program product obtain the attribute information of the backup cluster corresponding to the main cluster when the main cluster fails; export the historical data in the main cluster, and transmit the historical data to the backup cluster corresponding to the attribute information according to the attribute information, and obtain the real-time data in the main cluster, and transmit the real-time data to the backup cluster; use the backup cluster as the new data source of the secondary cluster, take over the service of the main cluster, and perform real-time data transmission to the secondary cluster. By adopting this method, by switching the main cluster and the backup cluster in the cluster dimension, when all nodes of the main cluster are down, the high availability of the data warehouse can be achieved, and the stability of the data warehouse can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 An application environment diagram of a data migration method in an embodiment;

[0055] Figure 2 A schematic diagram of a data migration method in one embodiment;

[0056] Figure 3 A flowchart of steps for determining the state of a primary cluster in one embodiment;

[0057] Figure 4 A schematic diagram of the process of historical data migration and real-time data migration in one embodiment;

[0058] Figure 5 A schematic diagram of a process of switching a data source in one embodiment;

[0059] Figure 6 A flowchart of the backup steps of the main cluster in a normal state in an embodiment

[0060] Figure 7 is a structural block diagram of a data migration device in one embodiment;

[0061] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0063] The data migration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in the figure, the main cluster terminal 102 communicates with the standby cluster 104 and the secondary cluster 106 through the network. When the main cluster fails, the main cluster terminal 102 obtains the attribute information of the standby cluster 104 corresponding to the main cluster; the main cluster terminal 102 exports the historical data in the main cluster, transmits the historical data to the standby cluster 104 corresponding to the attribute information according to the attribute information, and obtains the real-time data in the main cluster, and transmits the real-time data to the standby cluster 104; the main cluster terminal 102 uses the standby cluster 104 as the new data source of the secondary cluster 106, takes over the main cluster service, and performs real-time data transmission to the secondary cluster 106.

[0064] In an exemplary embodiment, Figure 2 As shown, a data migration method is provided, which is applied to Figure 1 Taking the master cluster terminal 102 in FIG. 1 as an example, the method includes the following steps 202 to 206. Among them:

[0065] Step 202: When a failure occurs in the active cluster, obtain attribute information of the standby cluster corresponding to the active cluster.

[0066] In an embodiment of the present application, the main cluster is a data warehouse based on MPP (Massively Parallel Processing) technology, and the main cluster may include multiple database nodes. When a failure occurs in the main cluster, the operation and maintenance personnel can manually start the master-slave switching of the main cluster, and input the attribute information of the standby cluster corresponding to the main cluster, and the terminal of the main cluster can then obtain the attribute information of the standby cluster corresponding to the main cluster. For example, the operation and maintenance personnel can obtain the attribute information of the standby cluster through configuration files, management tools, or command line interfaces, and the attribute information may include the unique identifier of the standby cluster, network configuration information (such as IP address, port number), health status, etc. Then, the operation and maintenance personnel input the attribute information of the standby cluster into the terminal of the main cluster, and the terminal can determine the standby cluster as the new main cluster.

[0067] In an optional embodiment, there may be multiple standby clusters, and the terminal may determine the target standby cluster based on the attribute information of each standby cluster. For example, the terminal may determine the standby cluster based on the configuration and performance of the standby cluster, including parameters such as hardware configuration, storage capacity, processing power, and network bandwidth, to ensure that the standby cluster has sufficient performance and resources to bear the workload when the main cluster fails; the terminal may also ensure that the connection between the standby cluster and the main cluster is stable and reliable, and the network delay is low, based on the network connection between the main cluster and the standby cluster, so as to perform data synchronization and switching operations in a timely manner.

[0068] In an optional embodiment, the terminal may autonomously start the active / standby switching according to the relationship between the delay of data transmission in the active cluster and a preset delay threshold.

[0069] Step 204 : export the historical data in the active cluster, transmit the historical data to the standby cluster corresponding to the attribute information according to the attribute information, obtain the real-time data in the active cluster, and transmit the real-time data to the standby cluster.

[0070] In an embodiment of the present application, when the delay of the main cluster exceeds the preset tolerance range or the main cluster is about to have a long-term shutdown operation, the terminal can actively or passively perform the master-slave switch, and perform the master-slave switch by connecting the terminal to the self-service platform, wherein the self-service platform is the real-time operation and maintenance program of the main cluster and the standby cluster, and can start or stop the master-slave switch for the main cluster and the standby cluster, or perform backtracking operations, etc. The terminal exports historical data from the main cluster according to a preset data export tool or according to a query command, and completes the migration of real-time data after completing the export of historical data, wherein historical data migration refers to the process in which the terminal migrates all historical data in the main cluster before the switching moment during the switching process; real-time data migration refers to the process in which the standby cluster needs to keep data consistent with the main cluster in real time after the switching is completed, and the terminal performs real-time data backtracking and migrates the real-time data to the standby cluster.

[0071] Specifically, when the terminal performs the master-slave switching between the main cluster and the backup cluster, the attribute information such as the unique identifier or network configuration information of the main cluster in the self-service platform can be updated to the attribute information corresponding to the backup cluster to achieve the master-slave switching. In the case of multiple backup clusters, the terminal determines the backup cluster to connect to based on the service availability of the backup cluster, that is, only when the backup cluster is in an available state will the main cluster use it as the target for connecting to the backup.

[0072] In an optional embodiment, the primary cluster receives real-time data from Kafka (a distributed streaming platform). Both the primary cluster and the backup cluster include a real-time program, which is used to consume Kafka data and write it into the database in real time. After the primary cluster transfers historical data to the backup cluster, the primary cluster shuts down the real-time data and instructs the backup cluster to start the real-time program to receive Kafka data in real time.

[0073] Step 206 , the standby cluster is used as a new data source of the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster.

[0074] In an embodiment of the present application, the terminal switches the standby cluster to a new data source, and ensures that the secondary cluster is connected to the standby cluster by updating the connection settings or configuration files between the primary cluster and the secondary cluster, so as to obtain the real-time data of the standby cluster, and enable the standby cluster to take over the services of the primary cluster and perform real-time data transmission to the secondary cluster. Optionally, for real-time data transmission, database replication can be used. If the primary cluster and the secondary cluster use the database as data storage, a replication mechanism between the primary and standby databases can be configured to achieve real-time data transmission; message queue, using a message queue system such as Kafka, etc., to send the real-time data generated by the primary cluster to the secondary cluster for consumption and processing; based on network transmission: or, the real-time data generated by the primary cluster is copied to the secondary cluster through a network transmission tool.

[0075] In the above data migration method, by performing active / standby switching between the primary cluster and the standby cluster in the cluster dimension, when all nodes of the primary cluster are down, high availability of the data warehouse can be achieved, thereby improving the stability of the data warehouse.

[0076] In an exemplary embodiment, Figure 3 As shown, before step 202, the method further includes steps 302 to 304. Among them:

[0077] Step 302: Send a heartbeat signal to the main cluster and obtain it in real time.

[0078] In the embodiment of the present application, the terminal periodically sends heartbeat signals to the main cluster to monitor the operation status of the main cluster. These heartbeat signals can be network communication packets, specific data packets or similar mechanisms, which are used to confirm the survival status of the main cluster.

[0079] Step 304: Determine the delay information of the main cluster based on the heartbeat signal of the main cluster. When the delay information is greater than a preset threshold, display the delay information on a display page and send an alarm message.

[0080] In an embodiment of the present application, once a large amount of data delay is detected in the main cluster, the terminal will trigger a monitoring and alarm system. For example, the terminal determines the delay information of the main cluster by receiving the heartbeat signal of the main cluster. When the delay information is greater than a preset threshold, the terminal can display relevant monitoring information on a visualization page, including delay conditions, main cluster status, etc., and display it intuitively in the form of charts, indicators, etc., so that relevant personnel can monitor the cluster situation at any time. Alternatively, alarm information can be sent to relevant personnel in a configured manner, such as through emails, text messages, instant messaging tools and other channels, to promptly inform them of the operating status of the main cluster and possible delays.

[0081] In this embodiment, when a large amount of data delay occurs, a combination of visual page monitoring and sending alarm information is used to ensure that relevant personnel can promptly discover and respond to cluster conditions, thereby ensuring the normal operation of the cluster and business continuity.

[0082] In an exemplary embodiment, Figure 4 As shown, step 204 includes steps 402 to 404. Among them:

[0083] Step 402 : export the historical data in the active cluster according to a preset data format, determine the standby cluster according to the identifier in the attribute information, and transmit the exported historical data to the standby cluster according to the preset data format.

[0084] In an embodiment of the present application, before executing the historical data migration, the terminal needs to back up the historical data of the main cluster to prevent data loss during the data migration process. Then, the terminal exports the historical data in the main cluster according to the preset data format through the database backup tool, and determines the standby cluster for receiving the historical data based on the identifier of the acquired attribute information of the standby cluster, wherein the terminal can determine the unique identifier of the standby cluster based on the mapping relationship between the identifier value and the standby cluster. The terminal transmits the exported historical data to the standby cluster through network transmission, storage device transmission, etc. Optionally, after the terminal completes the migration of historical data, the terminal can verify the exported historical data, for example, by verifying the accuracy of the exported historical data through a data verification tool or a comparison algorithm.

[0085] Step 404: Acquire real-time data, store the real-time data in a preset storage space, and instruct the standby cluster to start a real-time program.

[0086] The real-time program is used for the standby cluster to receive real-time data in real time.

[0087] In the embodiment of the present application, the terminal obtains the real-time data uploaded by the external service in real time, and stores the real-time data in the preset storage space of the main cluster. After the main cluster completes the migration of historical data, the terminal migrates the real-time data in the preset storage space and instructs the standby cluster to start the real-time program to ensure that the standby cluster can receive the latest updated data from the main cluster in real time. Optionally, during the real-time data migration process, the terminal needs to continuously confirm the consistency of the standby cluster data with the main cluster data to ensure that the data on the standby cluster can be updated in real time and keep in sync with the main cluster.

[0088] In this embodiment, by migrating historical data and real-time data, data consistency between the active cluster and the standby cluster can be guaranteed, and the standby cluster can be ensured to update data in real time, thereby achieving active-standby switching of the active cluster and improving the stability of the data warehouse.

[0089] In an exemplary embodiment, Figure 5 As shown, step 206 includes steps 502 to 504. Among them:

[0090] Step 502: Update the first connection information of the secondary cluster corresponding to the primary cluster to the second connection information corresponding to the standby cluster.

[0091] In an embodiment of the present application, after the master-slave switching is completed between the host group and the backup cluster, the configuration information of the backup cluster needs to be updated to ensure that the secondary cluster can use the backup cluster as the new primary cluster. For example, the terminal updates the cluster configuration file of the backup cluster, and modifies the first connection information of the secondary cluster, and updates the first connection information to the second connection information of the backup cluster.

[0092] Step 504: instruct the standby cluster to synchronize data with the secondary cluster, and complete the migration of service traffic from the primary cluster to the standby cluster.

[0093] In the embodiment of the present application, the terminal instructs the standby cluster to synchronize data with the secondary cluster. The terminal updates the configuration information of the secondary cluster and uses the standby cluster as the new primary cluster corresponding to the secondary cluster to synchronize data. The terminal gradually switches the service traffic in the primary cluster to the standby cluster. That is, after the master-slave switching between the primary cluster and the standby cluster is completed, the standby cluster becomes the replication source of the secondary cluster, and the secondary cluster ensures service continuity by receiving the service data of the standby cluster.

[0094] In this embodiment, through real-time data processing technology and redundant backup technology, the data source of the secondary cluster is switched from the main cluster to the secondary cluster, meeting the secondary cluster's demand for real-time data processing, ensuring the high availability of the data warehouse, and thereby improving the stability of the data warehouse.

[0095] In an exemplary embodiment, the method further comprises:

[0096] When the primary cluster returns to normal operation, the real-time data is synchronized back to the primary cluster, and the primary cluster is used as the data source to transmit real-time data to the secondary cluster.

[0097] In the embodiment of the present application, after the master-slave switch is completed, the standby cluster can continue to complete data reception and data processing, and continue to provide business services. After the administrator investigates and repairs the unavailability of the master cluster, the unavailable master cluster is restored to the normal master cluster state, that is, when the operation state of the master cluster is normal, since the equipment performance of the master cluster is generally higher than that of the standby cluster, the terminal switches the data source of the secondary cluster back to the master cluster, synchronizes the real-time data back to the master cluster, and performs read and write operations on the business data according to the master cluster.

[0098] In this embodiment, when the state of the main cluster is restored to an available state, the performance of data processing can be improved by synchronizing the real-time data back to the main cluster and receiving the real-time data and processing the data through the main cluster.

[0099] In an exemplary embodiment, Figure 6 As shown, before step 202, the method further includes steps 602 to 604. Among them:

[0100] Step 602: When the primary cluster operates normally, all data of the primary cluster is backed up to the standby cluster according to a first preset period.

[0101] In the embodiment of the present application, when the main cluster is operating normally, the main cluster data is updated in real time, and the data is backed up to the standby cluster through a preset replication tool. When the main cluster fails or is shut down for a long time, the standby cluster can take over and continue to provide services. In order to achieve rapid backup and recovery of data, when the main cluster is operating normally, the terminal backs up the full data to the standby cluster according to the first preset period, wherein the first preset period can be a quarterly or annual period. Optionally, after the terminal completes the backup of the full data, in the subsequent full backup, the historical backup data can be deleted after the backup is completed.

[0102] Step 604: Back up the incremental data of the primary cluster to the standby cluster according to a second preset period.

[0103] In the embodiment of the present application, the second preset period is smaller than the first preset period, and the second preset period may be based on days or hours. After completing the first full data backup, the terminal backs up the incremental data according to the second preset period.

[0104] In this embodiment, by performing a full backup once and periodic incremental backups, the backup time and storage space can be reduced, thereby improving the backup efficiency.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a data migration device for implementing the data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data migration device embodiments provided below can refer to the limitations on the data migration method above, and will not be repeated here.

[0107] In an exemplary embodiment, Figure 7 As shown, a data migration device 700 is provided, comprising: an acquisition module 701, a first transmission module 702 and a second transmission module 703, wherein:

[0108] The acquisition module 701 is used to acquire the attribute information of the standby cluster corresponding to the main cluster when the main cluster fails;

[0109] The first transmission module 702 is used to export the historical data in the active cluster, transmit the historical data to the standby cluster corresponding to the attribute information according to the attribute information, obtain the real-time data in the active cluster, and transmit the real-time data to the standby cluster;

[0110] The second transmission module 703 is used to use the standby cluster as a new data source for the secondary cluster, take over the service of the primary cluster, and perform real-time data transmission to the secondary cluster.

[0111] In one embodiment, the apparatus 700 further includes:

[0112] The sending module is used to send heartbeat signals to the main cluster and obtain them in real time;

[0113] The alarm module is used to determine the delay information of the main cluster based on the heartbeat signal of the main cluster. When the delay information is greater than a preset threshold, the delay information is displayed on a display page and an alarm message is sent.

[0114] In one embodiment, the first transmission module 702 is specifically used to export the historical data in the primary cluster according to a preset data format, determine the standby cluster according to the identifier in the attribute information, and transmit the exported historical data to the standby cluster according to the preset data format;

[0115] Acquire real-time data, store the real-time data in a preset storage space, and instruct the standby cluster to start a real-time program; the real-time program is used for the standby cluster to receive real-time data in real time.

[0116] In one embodiment, the second transmission module 703 is specifically used to update the first connection information of the secondary cluster corresponding to the primary cluster to the second connection information corresponding to the standby cluster;

[0117] Instruct the standby cluster to synchronize data with the secondary cluster to complete the migration of service traffic from the primary cluster to the standby cluster.

[0118] In one embodiment, the apparatus 700 further includes:

[0119] The third transmission module is used to synchronize the real-time data back to the main cluster when the main cluster recovers to a normal operating state, and use the main cluster as a data source to perform real-time data transmission to the secondary cluster.

[0120] In one embodiment, the apparatus 700 further includes:

[0121] A first backup module, configured to back up all data of the primary cluster to the backup cluster according to a first preset period when the primary cluster operates normally;

[0122] The second backup module is used to back up the incremental data of the primary cluster to the standby cluster according to a second preset period.

[0123] Each module in the above data migration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0124] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store historical data and real-time data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented.

[0125] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0127] When a main cluster fails, obtaining attribute information of a backup cluster corresponding to the main cluster;

[0128] Exporting historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the primary cluster, and transmitting the real-time data to the standby cluster;

[0129] The standby cluster is used as a new data source for the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster.

[0130] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0131] Send heartbeat signals to the main cluster and obtain them in real time;

[0132] The delay information of the main cluster is determined based on the heartbeat signal of the main cluster. When the delay information is greater than a preset threshold, the delay information is displayed on a display page and an alarm message is sent.

[0133] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0134] Exporting the historical data in the primary cluster according to a preset data format, determining a standby cluster according to the identifier in the attribute information, and transmitting the exported historical data to the standby cluster according to the preset data format;

[0135] Real-time data is acquired, the real-time data is stored in a preset storage space, and the standby cluster is instructed to start a real-time program; the real-time program is used for the standby cluster to receive the real-time data in real time.

[0136] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0137] Updating the first connection information of the secondary cluster corresponding to the primary cluster to the second connection information corresponding to the standby cluster;

[0138] The standby cluster is instructed to synchronize data with the secondary cluster, and service traffic migration from the primary cluster to the standby cluster is completed.

[0139] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0140] When the main cluster is restored to a normal operating state, the real-time data is synchronized back to the main cluster, and the main cluster is used as a data source to perform real-time data transmission to the secondary cluster.

[0141] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0142] When the main cluster operates normally, backing up all data of the main cluster to the standby cluster according to a first preset period;

[0143] According to a second preset period, the incremental data of the primary cluster is backed up to the standby cluster.

[0144] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0145] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0148] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data migration method, characterized in that: The method comprises: When a main cluster fails, obtain the attribute information of the backup cluster corresponding to the main cluster; Exporting historical data in the primary cluster, transmitting the historical data to a standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the primary cluster, and transmitting the real-time data to the standby cluster; The standby cluster is used as a new data source of the secondary cluster, takes over the service of the primary cluster, and performs real-time data transmission to the secondary cluster; The main cluster and the standby cluster both include real-time programs, and the exporting of historical data in the main cluster, transmitting the historical data to the standby cluster corresponding to the attribute information according to the attribute information, acquiring real-time data in the main cluster, and transmitting the real-time data to the standby cluster include: Exporting the historical data in the primary cluster according to a preset data format, determining a standby cluster according to the identifier in the attribute information, and transmitting the exported historical data to the standby cluster according to the preset data format; Acquire real-time data, and store the real-time data in a preset storage space; After the historical data is transmitted to the standby cluster, the real-time program of the active cluster is shut down and the standby cluster is instructed to start the real-time program; the real-time program is used for the standby cluster to receive the real-time data in real time.

2. The method according to claim 1, characterized in that When the primary cluster fails, before obtaining the attribute information of the standby cluster corresponding to the primary cluster, the method further includes: Send heartbeat signals to the main cluster and obtain them in real time; The delay information of the main cluster is determined based on the heartbeat signal of the main cluster. When the delay information is greater than a preset threshold, the delay information is displayed on a display page and an alarm message is sent.

3. The method according to claim 1, characterized in that: The method of using the standby cluster as a new data source of the secondary cluster, taking over the service of the primary cluster, and performing real-time data transmission to the secondary cluster includes: Updating the first connection information of the secondary cluster corresponding to the primary cluster to the second connection information corresponding to the standby cluster; The standby cluster is instructed to synchronize data with the secondary cluster, and service traffic migration from the primary cluster to the standby cluster is completed.

4. The method according to claim 1, characterized in that: The method further comprises: When the main cluster is restored to a normal operating state, the real-time data is synchronized back to the main cluster, and the main cluster is used as a data source to perform real-time data transmission to the secondary cluster.

5. The method according to any one of claims 1 to 4, characterized in that: When the primary cluster fails, before obtaining the attribute information of the standby cluster corresponding to the primary cluster, the method further includes: When the main cluster operates normally, backing up all data of the main cluster to the standby cluster according to a first preset period; According to a second preset period, the incremental data of the primary cluster is backed up to the standby cluster.

6. A data migration device, characterized in that: The device comprises: An acquisition module, used to acquire attribute information of a standby cluster corresponding to the main cluster when a main cluster fails; A first transmission module, configured to export the historical data in the primary cluster, transmit the historical data to a standby cluster corresponding to the attribute information according to the attribute information, obtain the real-time data in the primary cluster, and transmit the real-time data to the standby cluster; A second transmission module is used to use the standby cluster as a new data source of the secondary cluster, take over the service of the primary cluster, and perform real-time data transmission to the secondary cluster; The primary cluster and the standby cluster both include real-time programs, and the first transmission module is specifically used to export historical data in the primary cluster according to a preset data format, determine the standby cluster according to the identifier in the attribute information, and transmit the exported historical data to the standby cluster according to the preset data format; Acquire real-time data, and store the real-time data in a preset storage space; After the historical data is transmitted to the standby cluster, the real-time program of the active cluster is shut down and the standby cluster is instructed to start the real-time program; the real-time program is used for the standby cluster to receive the real-time data in real time.

7. The device according to claim 6, characterized in that The device also includes: The sending module is used to send heartbeat signals to the main cluster and obtain them in real time; The alarm module is used to determine the delay information of the main cluster based on the heartbeat signal of the main cluster, and when the delay information is greater than a preset threshold, the delay information is displayed on a display page and an alarm message is sent.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images