Data migration method, device, computer equipment and storage medium

Abstract

The embodiment of the present application belongs to the field of big data, and relates to a data migration method, including: establishing a connection between a data migration scheduling center and a source relational database and a target database cluster, and matching the static information of the two databases; detecting whether there is data to be migrated in each database of the target database cluster; if so, performing data migration based on the existing data cache and the database connection configuration information stored by the data migration scheduling center; otherwise, if it is detected that there is data to be migrated in the source relational database, based on the static information of the source relational database and the target database cluster, the corresponding data migration strategy is mobilized according to different data types to perform data migration. The present application also provides a data migration device, a computer device, and a storage medium. The present application can uniformly manage database static information, and selecting different data migration strategies is conducive to accelerating data synchronization progress, increasing migration throughput, and improving data migration efficiency.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data migration method, device, computer equipment and storage medium. Background Art

[0002] In the prior art, monitoring software, as a type of IoT monitoring software, can collect real-time data from smart devices, and the data it generates is linearly related to time. Usually, according to the needs of data analysis, the device data to be collected is very dense. The traditional technology approach is to store data in a relational database and use it as a source for subsequent data analysis. However, traditional databases have very limited capabilities for analyzing, processing, and storing large amounts of time series data, and their performance requirements for server hardware resources are very high. In the face of big data scenarios, traditional data repositories are difficult to implement big data analysis and retrieval.

[0003] Therefore, a new type of NewSQL database is introduced as a data warehouse for analysis and storage, but it is necessary to migrate the collected data from the traditional relational database to the new NewSQL database. The existing technology is to gradually batch query the source data into the memory in a sorting and query paging mode, and then perform syntax conversion and insert it into the NewSQL database. However, the scope of effect is very narrow and is only suitable for the migration of a small amount of data. For tens of millions of data, the paging query becomes slower towards the end of the data. It can be seen that when there is a large amount of data in the monitoring software that needs to be migrated to the database, the migration method provided by the existing technology has the problems of slow migration speed and low efficiency. Summary of the invention

[0004] The purpose of the embodiment of the present application is to propose a data migration method to solve the problem of slow migration speed and low efficiency when there is a large amount of data in the monitoring software that needs to be migrated to the database, and at the same time solve the problem of inconsistency between the fields of the source data and the target data when migrating big data.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a data migration method, which is applicable to a data migration system of smart devices. The system includes a data migration scheduling center, a source relational database, and a target database cluster, and adopts the following technical solutions:

[0006] Establishing a connection between the data migration scheduling center and the source relational database and the target database cluster, and matching the source relational database and the target database cluster according to static information of the source relational database and the target database cluster, wherein the source relational database includes data to be migrated;

[0007] Detecting whether the data cache to be migrated already exists in the target database cluster;

[0008] If the target database cluster has the data cache to be migrated, then based on the existing data cache and the database connection configuration information stored in the data migration scheduling center, directly pull the data to be migrated from the database where the data to be migrated exists to the database where the data has not been migrated;

[0009] If the target database cluster does not have the data cache to be migrated, detecting whether the source relational database has the data to be migrated;

[0010] If the data to be migrated exists in the source relational database, based on the static information of the source relational database and the target database cluster, the corresponding data migration strategy is mobilized according to different data types to migrate the data to be migrated from the source relational database to the corresponding target database in the target database cluster, and a migrated data cache is established in the target database.

[0011] Furthermore, the static information includes data table mapping configuration information, data field mapping configuration information and database connection configuration information, and the steps of establishing a connection between the data migration scheduling center and the source relational database and the target database cluster, and matching according to the static information of the source relational database and the target database cluster include:

[0012] Sending a first connection request signal to the source relational database through the data migration scheduling center, and sending a second connection request signal to the target database cluster;

[0013] Establishing a network connection between the source relational database and the target database cluster according to a first verification result returned by the source relational database and a second verification result returned by the target database cluster; and

[0014] According to the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information, the data tables and data fields of the source relational database and the target database cluster are matched and mapped.

[0015] Further, based on the static information of the source relational database and the target database cluster, the corresponding data migration strategy is mobilized according to different data types, the data to be migrated is migrated from the source relational database to the corresponding target database in the target database cluster, and the step of establishing a migrated data cache in the target database specifically includes:

[0016] Simulating a source relational database client through the data migration scheduling center and sending a data query request to the source relational database;

[0017] Simulating a target database client through the data migration scheduling center and sending a data write request to the target database;

[0018] According to the data query connection result returned by the source relational database, the data write connection result returned by the target database, the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information, the corresponding data migration strategy is mobilized based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster, and a migrated data cache is established in the target database.

[0019] Furthermore, the data types include first-class data, second-class data, and third-class data with increasing data volumes, and the step of mobilizing corresponding data migration strategies based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster specifically includes:

[0020] Read and determine the data types corresponding to the number of data items corresponding to the data to be migrated in each data table of the source relational database;

[0021] If the data type of the number of data items corresponding to the data to be migrated is the first type of data, then based on the preset number of unit data items, the data is polled and read from the source relational database to the data migration scheduling center, and then sent to the target database corresponding to the target database cluster;

[0022] If the data type of the data items corresponding to the data to be migrated is the second type of data, the data table partition of the source relational database is split into multiple data sub-tables, and the multiple data sub-tables are sent to the target database corresponding to the target database cluster in parallel, and then the data is merged according to the data field mapping configuration information;

[0023] If the data type of the number of data items corresponding to the data to be migrated is the three types of data, the data migration scheduling center simulates the source relational database client, parses the underlying data transmission protocol of the source relational database, reads the data to be migrated and synchronizes it to the target database corresponding to the target database cluster, and then merges the data according to the data field mapping configuration information.

[0024] Furthermore, the step of simulating the source relational database client through the data migration scheduling center, parsing the underlying data transmission protocol of the source relational database, reading the data to be migrated and synchronizing it to the target database corresponding to the target database cluster for data merging specifically includes:

[0025] Based on the data field mapping configuration information, the target data table to be synchronized by the target database is split into multiple sub-data tables, and the sub-data tables are sent to the source relational database to obtain multiple data tables to be migrated corresponding to the multiple sub-data tables;

[0026] Calling the calling interface of the data migration scheduling center, controlling the target database to issue a data transmission command to disguise as the source relational database client to parse the underlying data transmission protocol of the source relational database, and reading the multiple data tables to be migrated by streaming data reading;

[0027] The data of the multiple data tables to be migrated that are read are merged.

[0028] Furthermore, after the step of mobilizing the corresponding data migration strategy according to different data types based on the static information of the source relational database and the target database cluster, and migrating the data to be migrated from the source relational database to the target database corresponding to the target database cluster, the method further includes:

[0029] Record the number of data items remaining to be migrated in the source relational database and the number of migrated data items that have been migrated to the target database;

[0030] The current synchronization progress is calculated according to the number of remaining data items to be migrated and the number of migrated data items, and a corresponding data interface is selected according to the current synchronization progress for a third party to read data.

[0031] In order to solve the above technical problems, the embodiment of the present application further provides a data migration device, which is applicable to a data migration system of an intelligent device. The system includes a data migration scheduling center, a source relational database, and a target database cluster, and adopts the following technical solution:

[0032] A connection module, used to establish a connection between the data migration scheduling center and the source relational database and the target database cluster, and to match the source relational database and the target database cluster according to static information of the source relational database and the target database cluster, wherein the source relational database includes data to be migrated;

[0033] A first detection module, used to detect whether the data cache to be migrated already exists in the target database cluster;

[0034] A first scheduling module is used for directly pulling the data to be migrated from the database where the data to be migrated exists to the database where the data has not been migrated, based on the existing data cache and the database connection configuration information stored in the data migration scheduling center, if the target database cluster has the data cache to be migrated;

[0035] A second detection module, configured to detect whether the source relational database has the data to be migrated if the target database cluster does not have the data cache to be migrated;

[0036] The second scheduling module is used to migrate the data to be migrated from the source relational database to the corresponding target database in the target database cluster based on the static information of the source relational database and the target database cluster and according to different data types, if the data to be migrated exists in the source relational database, and establish a migrated data cache in the target database.

[0037] Further, the static information includes data table mapping configuration information, data field mapping configuration information and the database connection configuration information, and the connection module includes:

[0038] A first request submodule, configured to send a first request connection signal to the source relational database through the data migration scheduling center, and send a second request connection signal to the target database cluster;

[0039] A connection submodule, configured to establish a network connection between the source relational database and the target database cluster according to a first verification result returned by the source relational database and a second verification result returned by the target database cluster;

[0040] A mapping submodule is used to match and map the data tables and data fields of the source relational database and the target database cluster according to the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information.

[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0042] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of a data migration method described in any embodiment are implemented.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0044] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of a data migration method described in any embodiment are implemented.

[0045] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: In the embodiments of the present invention, by providing a data migration scheduling center, it is possible to achieve unified management of the static information of the source relational database and the target database cluster, and solve the problem of inconsistent fields between the data in the source relational database and the data in the target database during big data migration; secondly, when the target database cluster already has a cache of data to be migrated, it can be based on the existing cache of data to be migrated and the database connection configuration information stored in the data migration scheduling center, and directly pull the data to be migrated from the database where the data to be migrated exists to the database where the data has not been migrated, that is, when the target database cluster, it is possible to directly pull database data from each other, avoiding reading from the source relational database every time, which is conducive to accelerating the synchronization progress of each database in the target database cluster. In addition, when it is necessary to migrate the data to be migrated from the source relational database to the corresponding database of the target database cluster, different data migration strategies can be selected based on the static information of the source relational database and the target database cluster according to the data type of the data to be migrated to perform data synchronization migration, which is conducive to achieving the maximum throughput of migration and improving data migration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0048] Figure 2 is a flow chart of an embodiment of a data migration method according to the present application;

[0049] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S1;

[0050] Figure 4 yes Figure 2 A flowchart of a specific implementation of step S4;

[0051] Figure 5 yes Figure 4 A flowchart of a specific implementation of step S43;

[0052] Figure 6 yes Figure 5 A flowchart of a specific implementation of step S434;

[0053] Figure 7 is a flow chart of another embodiment of a data migration method according to the present application;

[0054] Figure 8 is a structural schematic diagram of an embodiment of a data migration device according to the present application;

[0055] Fig. 9 yes Figure 8 A schematic structural diagram of a specific implementation of the connection module shown;

[0056] Fig.10 yes Figure 8 A structural schematic diagram of a specific implementation of the second scheduling module shown;

[0057] Fig.11 yes Fig.10 A schematic diagram of the structure of a specific implementation of the scheduling submodule shown;

[0058] Fig.12 is a structural schematic diagram of another embodiment of a data migration device according to the present application;

[0059] Fig.13 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0063] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0064] The terminal devices 101, 102, 103 interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications are installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0066] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0067] It should be noted that a data migration method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, a data migration device is generally arranged in the server / terminal device.

[0068] It should be understood that Figure 1 The number of terminal devices 101, 102, 103, networks 104 and servers 105 in the embodiment is only for illustration. Any number of terminal devices 101, 102, 103, networks 104 and servers 105 may be provided according to implementation requirements.

[0069] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for data migration according to the present application. The above-mentioned method for data migration includes the following steps:

[0070] Step S1, establishing a connection between the data migration scheduling center and the source relational database and the target database cluster, and matching the source relational database and the target database cluster according to static information of the source relational database and the target database cluster.

[0071] In this embodiment, a data migration method is executed on an electronic device (eg Figure 1 The server / terminal device shown in the figure can receive data access requests sent by external clients through wired connection or wireless connection. It should be noted that the above wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0072] Among them, the data migration scheduling center includes the static information of the source relational database and the target database cluster (database connection configuration information, including (database type, IP address, IP port, database user name, database password), data table mapping configuration information, data field mapping configuration information, etc.), which can realize the maintenance and management of the source relational database and the target database cluster, and has an internal synchronization algorithm. Different data migration strategies can be selected for data migration according to different types. In the data migration scheduling center, matching and mapping can be performed based on the static information of the source relational database and the target database cluster. Specifically, it can include matching and mapping the data table mapping configuration information and data field mapping configuration information of the source relational database with the data table mapping configuration information of the target database cluster in the data migration scheduling center, and there is a corresponding relationship between the data tables and data fields of the two.

[0073] The above-mentioned source relational database includes the data to be migrated. The source relational database has high requirements on the server's CPU performance, memory space, disk space and other hardware resources. In the context of big data, the source relational database is often used as a data warehouse and can be used to store data collected in real time by smart devices. Smart devices can include monitoring devices, which can be set up in shopping malls, roads, residences, office buildings and other areas.

[0074] The above-mentioned target database cluster includes a target database (target NewSQL database), which is the abbreviation for various new scalable / high-performance databases. The target database cluster not only has the storage and management capabilities of NoSQL for massive data, but also maintains the characteristics of traditional databases supporting ACID and SQL. The target database cluster includes at least one sub-database, that is, the target database cluster can be configured as a cluster and have multiple sub-databases. Even with a data volume of hundreds of billions, data analysis and storage can also be quickly realized. Especially for monitoring software devices, monitoring data is generated every minute and every second. An ordinary data center can generate nearly tens of billions of historical data every year. After the data is collected and stored for several years, it is generally around hundreds of billions of data. In this case, the introduction of the above-mentioned target database cluster can not only realize storage, but also provide data analysis.

[0075] Specifically, because the data migration scheduling center has already obtained the static information of the source relational database and the target database cluster in advance, it can send requests to the source relational database and the target database cluster respectively based on the static information, and connect to the source relational database and the target database cluster according to the feedback results. After the database connection is successful, the data tables and data fields of the source relational database and the target database cluster can be matched and mapped internally based on the static information, so that the data tables that need to be migrated synchronously can be decomposed into multiple small data tables according to the data fields.

[0076] Step S2: Detect whether the target database cluster already has data cache to be migrated.

[0077] In this embodiment, before each migration, it is determined whether part of the data to be migrated in the source relational database has been synchronized and cached before (because each synchronization is stateless). If it is determined that some of the data to be migrated has been synchronized and cached / there is no synchronized and cached data in the data to be migrated, different data migration strategies can be adopted according to different situations.

[0078] Step S3: if the target database cluster has data cache to be migrated, based on the existing data cache and the database connection configuration information stored in the data migration scheduling center, directly pull the data to be migrated from the database with the data to be migrated to the database with the data not migrated.

[0079] In this embodiment, when it is detected that a database in the target database cluster already has a cache of data to be migrated or a cache of part of the data to be migrated, then based on the existing data cache and the database connection configuration information stored in the data migration scheduling center, the data can be directly pulled from the database including the data to be migrated to the database of the unmigrated data. For example: it is necessary to pull a certain amount of data to be migrated from the source relational database, and the data to be migrated includes data M. Before the migration, it is detected that the certain amount of data to be migrated / data M has been cached in database N1 before. At this time, other databases can directly pull the data to be migrated / data M from database N1.

[0080] In this way, it is possible to avoid pulling the data to be migrated from the source relational database again. Since relational databases only store and arrange data, they cannot compress data to reduce the size of data blocks to increase the data transmission rate. However, the data migration scheduling center can cache the synchronized data in NewSQL each time, reuse the data compression and data storage capabilities of NewSQL, and synchronize from the source relational database or from the data cache of the sub-database according to the order of synchronization, thereby improving the overall data synchronization efficiency.

[0081] Step S4: If the target database cluster does not have the data cache to be migrated, then detect whether the source relational database has the data to be migrated.

[0082] Specifically, when all databases in the target database cluster do not detect the above-mentioned data cache to be migrated, it is possible to detect in real time or periodically whether there is data to be migrated in the source relational database. If there is, migration can be performed; if not, real-time or periodic detection can continue.

[0083] Step S5: If there is data to be migrated in the source relational database, based on the static information of the source relational database and the target database cluster, the corresponding data migration strategy is mobilized according to different data types to migrate the data to be migrated from the source relational database to the corresponding target database in the target database cluster, and a migrated data cache is established in the target database.

[0084] Specifically, when there is data to be migrated in the source relational database, the data migration scheduling center can adopt different data migration strategies based on the static information of the source relational database and the target database cluster (data table mapping configuration information, data field mapping configuration information, and database connection configuration information) according to the data types of each data table in the source relational database, and migrate the data to be migrated in the source relational database to each data table in the target database cluster. The above data types may include multiple types, and the data migration strategies may correspond to the data types one by one, wherein the data types may be distinguished based on the number of data items, and divided into multiple data types according to the number of data items.

[0085] In an embodiment of the present invention, by providing a data migration scheduling center, it is possible to achieve unified management of the static information of the source relational database and the target database cluster, and solve the problem of inconsistent fields between the data in the source relational database and the data in the target database during big data migration; secondly, when the target database cluster already has a cache of data to be migrated, it is possible to directly pull the data to be migrated from the database where the data to be migrated exists to the database where the data has not been migrated based on the existing cache of data to be migrated and the database connection configuration information stored in the data migration scheduling center, that is, when the target database cluster, it is possible to directly pull database data from each other, avoiding reading from the source relational database every time, which is conducive to accelerating the synchronization progress of each database in the target database cluster. In addition, when it is necessary to migrate the data to be migrated from the source relational database to the corresponding database of the target database cluster, different data migration strategies can be selected based on the static information of the source relational database and the target database cluster according to the data type of the data to be migrated to perform data synchronization migration, which is conducive to achieving the maximum throughput of migration and improving data migration efficiency.

[0086] In some optional implementations of this embodiment, refer to Figure 3 As shown, Figure 3 FIG. 1 is a flowchart of a specific embodiment of step S1. Step S1 performed by the electronic device may specifically include:

[0087] Step S11 : sending a first connection request signal to a source relational database and sending a second connection request signal to a target database cluster through a data migration scheduling center.

[0088] Specifically, to establish a connection with the source relational database and the target database cluster, it is first necessary to send a request connection signal (first request connection signal) to the source relational database and a request connection signal (second connection signal) to the target database cluster through the data migration scheduling center to wait for feedback results from the source relational database and the target database cluster.

[0089] Step S12: establishing a network connection between the source relational database and the target database cluster according to the first verification result returned by the source relational database and the second verification result returned by the target database cluster.

[0090] Specifically, when the source relational database receives the first connection request signal, it can return the first verification result to the data migration scheduling center; and when the target database cluster receives the second request signal, it can return the second verification result to the data migration scheduling center. The above first verification result and the second verification result can indicate that the connection is agreed to be established. After receiving the feedback indicating that the connection is agreed to be established, the data migration scheduling center can establish a connection with the source relational database according to the static information of the source relational database, and establish a connection with the target database cluster according to the static information of the target database cluster.

[0091] Of course, if there is a connection failure, the data migration scheduling center can periodically send connection request signals to the source relational database and the target database cluster repeatedly, and stop sending request signals after reaching a preset time threshold or a preset number threshold.

[0092] Step S13, matching and mapping the data tables and data fields of the source relational database and the target database cluster according to the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information.

[0093] Specifically, the data migration scheduling center includes static information of the source relational database and the target database cluster, and the static information includes data table mapping configuration information, data field mapping configuration information, and data connection configuration information. After the connection is established, the data migration scheduling center matches and maps the data tables and data fields of the source relational database and the target database cluster internally according to the data table mapping configuration information, data field mapping configuration information, and database connection configuration information, so as to facilitate the subsequent decomposition of the data tables to be synchronized into multiple small data tables.

[0094] In this embodiment, the data migration scheduling center sends data request connection signals to the source relational database and the target database cluster respectively so as to connect them according to the feedback verification results, and then matches the parameters of the source relational database and the target database cluster to facilitate the subsequent partitioning of the data tables in the source relational database / target database cluster.

[0095] In some optional implementations, reference Figure 4 As shown, Figure 4 FIG. 5 is a flowchart of a specific embodiment of the above step S5. The step S5 performed by the above electronic device specifically includes:

[0096] Step S51: The data migration scheduling center simulates a source relational database client and sends a data query request to the source relational database.

[0097] Specifically, after establishing the connection and completing parameter matching, the data migration scheduling center can simulate / disguise itself as a source relational database client to send a data query request to the source relational database to request querying and reading the data to be migrated in the source relational database.

[0098] Step S52: The target database client is simulated by the data migration scheduling center, and a data write request is sent to the target database.

[0099] Specifically, the data migration scheduling center also simulates / disguises itself as a target database client to send a data write request to the target database, requesting that the subsequently read data to be migrated be migrated to the corresponding target database in the target database cluster, and establishes a migrated data cache in the target database, which can be used by subsequent real target database cluster clients to perform data query and analysis on the target database cluster.

[0100] Of course, if the sending of the data query request and / or the data write request times out, you can choose to resend it at a fixed time / a fixed number of times until a preset time threshold / number threshold is reached.

[0101] Step S53, based on the data query connection result returned by the source relational database, the data write connection result returned by the target database, the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information, the corresponding data migration strategy is mobilized based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster, and a migrated data cache is established in the target database.

[0102] Specifically, the above data query connection result can indicate the result of data query approval, and the above data write connection result can indicate the result of data write approval. When the source relational database agrees that the data migration scheduling center reads the data to be migrated, and the target database agrees that the data migration scheduling center writes the data to be migrated, the data migration scheduling center can synchronize the read data to be migrated to the target database based on the data table mapping configuration information and data field mapping configuration information of the source relational database and the target database. The target database will also provide real-time feedback on the synchronization status to the data migration scheduling center.

[0103] In this embodiment, by simulating the source relational database client and the target database cluster client respectively, the data to be migrated in the source relational database can be read and synchronized to the target database cluster.

[0104] In some optional implementations, reference Figure 5 As shown, Figure 5 FIG. 5 is a flowchart of a specific embodiment of the step S53. The step S53 performed by the electronic device specifically includes:

[0105] Step S531 , reading and determining the data type corresponding to the number of data items corresponding to the data to be migrated in each data table of the source relational database.

[0106] Specifically, before performing data migration, you can first read the data to be migrated in the source relational data and determine the number of data items to be migrated. According to the number of data items, the data can be divided into three data types, which can specifically include Class I data, Class II data, and Class III data with increasing data volumes, where Class I data can refer to data items within 500,000; Class II data can refer to data items within 10 million; and Class III data can refer to data items greater than 100 million. Of course, the above range is only a specific example of this embodiment, and can be flexibly adjusted within a certain range, for example, Class I data is within 600,000, Class II data is within 15 million, and so on. In this embodiment, this is not limited.

[0107] Step S532, if the data type of the data items corresponding to the data to be migrated is a type of data, then based on the preset unit data items, the data is polled and read from the source relational database to the data migration scheduling center, and then sent to the target database corresponding to the target database cluster.

[0108] Specifically, when it is determined that the number of data items to be migrated in each data table of the source relational database is within 500,000, the data migration scheduling center can adopt a polling, sorting and paging mode to poll and insert a preset unit number of 10,000 data items from the source relational database into the target database each time. The data migration scheduling center will enumerate all data items within 500,000 and perform concurrent data migration.

[0109] Step S533, if the data type of the data items corresponding to the data to be migrated is Class II data, the data table partition of the source relational database is split into multiple data sub-tables, and the multiple data sub-tables are sent to the target database corresponding to the target database cluster in parallel, and then the data is merged according to the data field mapping configuration information.

[0110] Specifically, when it is determined that the number of data entries to be migrated in each data table of the source relational database is within 10 million, the data migration scheduling center will send a logical statement to the source relational database, and the source relational database will split the data table with tens of millions of data into multiple data sub-tables according to its internal data partitions. The target database will then pull multiple data sub-tables concurrently, and finally merge the multiple pulled data sub-tables in the target database according to the data field mapping configuration information.

[0111] In this way, multiple data sub-tables can be pulled concurrently, and when data pulling errors occur due to poor network or other reasons, only the data of several erroneous sub-tables can be pulled in principle, rather than causing the entire data pulling to fail due to one data pulling error, which can reduce the overall error probability.

[0112] Step S534, if the data type of the number of data items corresponding to the data to be migrated is Category 3 data, the data migration scheduling center simulates the source relational database client, parses the underlying data transmission protocol of the source relational database, reads the data to be migrated and synchronizes it to the target database corresponding to the target database cluster, and then merges the data according to the data field mapping configuration information.

[0113] Specifically, when it is determined that the total number of data entries to be migrated in each data table of the source relational database is greater than 100 million, the source relational database client can be simulated through the data migration scheduling center to parse the underlying data transmission protocol of the source relational database to read the data to be migrated in each data table of the source relational database and synchronize it to the target database corresponding to the target database cluster, and then merge the data of each data table according to the data field mapping configuration information.

[0114] In this embodiment, data types are divided into three categories according to the number of data items, and each category adopts a different data migration method. In particular, when the number of data items reaches three types of data, the process of splitting the data table in the source relational database into data sub-tables according to data partitions will be very slow, because it also relies on data query capabilities. When the amount of data in most relational databases is at the level of three types of data, its processing efficiency is very low. At this time, it will take a long time to rely on its processing efficiency to perform data table partitioning. Therefore, adopting different data migration strategies according to different data types is effective in achieving the maximum throughput of migration.

[0115] In some optional implementations, reference Figure 6 As shown, Figure 6 FIG. 5 is a flowchart of a specific embodiment of the above step S534. The step S534 executed by the above electronic device specifically includes:

[0116] Step S5341: based on the data field mapping configuration information, split the target data table to be synchronized by the target database into multiple sub-data tables, and send them to the source relational database to obtain multiple data tables to be migrated corresponding to the multiple sub-data tables.

[0117] Specifically, the data migration scheduling center can split the target data tables (including data tables with billions of data volumes) required by the target database cluster into multiple target sub-data tables based on the data field mapping configuration information in the static information of the target database cluster. For example: split the target data table C into target sub-data table C1, target sub-data table C2, target sub-data table C3...target sub-data table Cn.

[0118] More specifically, after being split into multiple target sub-data tables, the multiple target sub-data tables can be sent to the source relational database through the data migration scheduling center, so that a unified data view is prepared in the source relational database based on the target sub-data tables, so as to construct multiple corresponding data tables to be migrated according to the target sub-data tables.

[0119] Step S5342, calling the calling interface of the data migration scheduling center, controlling the target database to issue a data transmission command to disguise as a source relational database client to parse the underlying data transmission protocol of the source relational database, and read multiple data tables to be migrated through streaming data reading.

[0120] Step S5343: merge the data of the multiple data tables to be migrated that have been read.

[0121] Specifically, by mobilizing the data migration API (Application Program Interface) inside the data migration scheduling center, the target database issues data transmission commands, disguises itself as a source relational database client, quotes the data transmission protocol of the source relational database, and uses streaming data reading to actively pull multiple split data tables to be migrated from the source relational database. After the data is pulled, through the high read and write performance of the target database, the data of each data table to be migrated is automatically spliced ​​according to the data mapping field information through logical statements, and integrated into the final complete data table in the target database. The above API is a calling interface left by the operating system to the application program. The application program makes the operating system execute the application program's commands by calling the operating system's API.

[0122] In this application, for the case where the data type belongs to the third category of data, the target data table library data table is first split and sent to the source relational database to prepare a unified data view, and then disguised as a source relational table for integration, thereby achieving rapid reading and migration of data at the billion level.

[0123] In some optional implementations, reference Figure 7 As shown, Figure 7 This is a flowchart of a specific embodiment provided in this embodiment. After the above step S5, the above electronic device may further perform the following steps:

[0124] Step S6, recording the number of data items remaining to be migrated in the source relational database and the number of data items that have been migrated to the target database.

[0125] Step S7, calculating the current synchronization progress according to the number of remaining data to be migrated and the number of migrated data, and selecting a corresponding data interface for a third party to read data according to the current synchronization progress.

[0126] Specifically, during the migration of data, the data migration scheduling center can record in real time the number of remaining data to be migrated in each data table of the source relational database and the number of migrated data that have been pulled from the target database. Then, the current synchronization progress can be calculated based on the total number of data to be migrated, the number of remaining data to be migrated, and the number of migrated data, and the corresponding data interface can be provided in real time for any third party to read the data.

[0127] This application calculates the data synchronization progress in real time and can provide a data interface in real time according to the progress for any third party to read the data.

[0128] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0129] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0130] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0131] Further references Figure 8 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a data migration device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0132] like Figure 8 As shown, a data migration device 800 of this embodiment includes: a connection module 801, a detection module 802, a first scheduling module 803 and a second scheduling module 805. Among them:

[0133] The connection module 801 is used to establish a connection between the data migration scheduling center and the source relational database and the target database cluster, and to match the source relational database and the target database cluster according to static information of the source relational database and the target database cluster, wherein the source relational database includes the data to be migrated;

[0134] A first detection module 802 is used to detect whether there is a cache of data to be migrated in the target database cluster;

[0135] The first scheduling module 803 is used to directly pull the data to be migrated from the database with the data to be migrated to the database with the data not migrated based on the existing data cache and the database connection configuration information stored in the data migration scheduling center if the target database cluster has the data cache to be migrated;

[0136] The second detection module 804 is used to detect whether the source relational database has data to be migrated if the target database cluster does not have data cache to be migrated;

[0137] The second scheduling module 805 is used to migrate the data to be migrated from the source relational database to the corresponding target database in the target database cluster based on the static information of the source relational database and the target database cluster and according to different data types, if there is data to be migrated in the source relational database, and to establish a migrated data cache in the target database.

[0138] In this embodiment, the data migration scheduling center has already acquired the static information of the source relational database and the target database cluster in advance, so the connection module 801 can send requests to the source relational database and the target database cluster respectively based on the static information, and perform database connection with the source relational database and the target database cluster according to the feedback results. After the database connection is successful, the data tables and data fields of the source relational database and the target database cluster can be matched and mapped internally based on the static information, so as to facilitate the subsequent decomposition of the data tables that need to be migrated synchronously into multiple small data tables according to the data fields.

[0139] More specifically, before each migration, the detection module 802 can be used to determine whether part of the data to be migrated in the source relational database has been synchronized before (because each synchronization is stateless). If it is determined that some of the data to be migrated has been synchronized / there is no synchronized data in the data to be migrated, different data migration strategies can be adopted according to different situations.

[0140] If the first detection module 802 detects the existence of data to be migrated or part of the data to be migrated in any database, the first scheduling module 803 can directly pull it from the database based on the data table mapping configuration information and data field mapping configuration information of the database. For example, it is necessary to pull a certain amount of data to be migrated from the source relational database, and the data to be migrated includes data M. Before the migration, it is detected that the certain amount of data to be migrated / data M has been cached in the database N1 before. At this time, other databases can directly pull the data to be migrated / data M from the database N1. In this way, it can be avoided to pull the data to be migrated from the source relational database again. Because the relational database only stores and arranges data, it is impossible to compress the data to reduce the size of the data block to increase the data transmission rate. However, the data migration scheduling center can cache the synchronized data in NewSQL each time, reuse the data compression capability and data storage capability of NewSQL, and synchronize from the source relational database or from the data cache of the sub-database according to the order of synchronization, thereby improving the overall data synchronization efficiency.

[0141] More specifically, when all databases in the target database cluster have not detected the above-mentioned data cache to be migrated, the second detection module 804 can detect in real time or periodically whether there is data to be migrated in the source relational database that needs to be migrated. If there is, migration can be performed; if not, real-time or periodic detection can continue.

[0142] More specifically, if the first detection module 802 does not detect the cache of data to be migrated in each database of the target database cluster, the second scheduling module 805 can adopt different data migration strategies according to the data types of each data table in the source relational database based on the data table mapping configuration information and data field mapping configuration information of the source relational database and the target database cluster, and migrate the data to be migrated in the source relational database to each data table of the target database cluster. The above data types may include multiple types, and the data migration strategies may correspond to the data types one by one, wherein the data types may be distinguished based on the number of data items, and divided into multiple data types according to the number of data items.

[0143] In an embodiment of the present invention, by providing a data migration scheduling center, it is possible to achieve unified management of the static information of the source relational database and the target database cluster. Secondly, when the detection module 802 detects that any database in the target database cluster already has a cache of data to be migrated, the first scheduling module 803 can directly pull the data to be migrated from the sub-database where the data to be migrated exists based on the data table mapping configuration information and data field mapping configuration information of the source relational database and the target database cluster. That is, when the target database cluster needs to configure the cluster to have multiple sub-databases, the sub-database data can be directly pulled from each other to avoid reading from the source relational database every time, which is conducive to accelerating the synchronization progress of multiple sub-databases in the target database cluster. In addition, when all databases in the target database cluster do not have a cache of data to be migrated, the second detection module 804 can be used to detect whether there is data to be migrated in the source relational database. If so, the second scheduling module 805 can be based on the data table mapping configuration information and data field mapping configuration information of the source relational database and the target database cluster, and select different data migration strategies according to the data type to which the data to be migrated belongs to perform data synchronization migration, which is conducive to achieving the maximum throughput of migration and improving data migration efficiency.

[0144] In some optional implementations of this embodiment, see Fig. 9 , is a structural diagram of a specific implementation of the connection module. Static information includes data table mapping configuration information, data field mapping configuration information and database connection configuration information. The connection module 801 includes a first request submodule 8011, a connection submodule 8012 and a mapping submodule 8013. Among them,

[0145] The first request submodule 8011 is used to send a first request connection signal to the source relational database through the data migration scheduling center, and send a second request connection signal to the target database cluster.

[0146] The connection submodule 8012 is used to establish a network connection between the source relational database and the target database cluster according to the first verification result returned by the source relational database and the second verification result returned by the target database cluster.

[0147] The mapping submodule 8013 is used to match and map the data tables and data fields of the source relational database and the target database cluster according to the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information.

[0148] In this embodiment, the first request submodule 8011 of the data migration scheduling center sends data request connection signals to the source relational database and the target database cluster respectively, so that the connection submodule 8012 connects according to the feedback verification results, and then the mapping submodule 8013 matches the parameters of the source relational database and the target database cluster, so as to facilitate the subsequent partitioning of the data tables in the source relational database / target database cluster.

[0149] In some optional implementations of this embodiment, see Fig.10 , is a structural diagram of a specific implementation of the second scheduling module. The second scheduling module 805 includes: a second request submodule 8051, a third request submodule 8052 and a scheduling submodule 8053. Among them,

[0150] The second request submodule 8051 is used to simulate the source relational database client through the data migration scheduling center and send a data query request to the source relational database.

[0151] The third request submodule 8052 is used to simulate the target database client through the data migration scheduling center and send a data write request to the target database.

[0152] The scheduling submodule 8053 is used to query the connection results returned by the source relational database, write the connection results returned by the target database, the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information, and mobilize the corresponding data migration strategy based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster, and establish a migrated data cache in the target database.

[0153] In this embodiment, the data migration scheduling center is simulated as a source relational database client, and a data query request is sent to the source relational database through the second request sub-module 8051; and the data migration scheduling center is simulated / disguised as a target database client, and a data write request is sent to the source relational database through the third request sub-module 8052, so that the scheduling sub-module 8053 can read the data to be migrated in the source relational database and synchronize the data to be migrated to the target database, and establish a migrated data cache in the target database.

[0154] In some optional implementations of this embodiment, see Fig.11 , which is a structural diagram of a specific implementation of the scheduling submodule. Among them, the data types include first-class data, second-class data and third-class data with increasing data volume, and the above-mentioned scheduling submodule 8053 includes: a reading subunit 80531, a first migration subunit 80532, a second migration subunit 80533, and a third migration subunit 80534. Among them,

[0155] The reading subunit 80531 is used to read and determine the data type corresponding to the number of data items corresponding to the data to be migrated in each data table of the source relational database.

[0156] The first migration subunit 80532 is used to poll and read the data from the source relational database to the data migration scheduling center based on a preset unit number of data if the data type of the data number corresponding to the data to be migrated is a type of data, and then send it to the target database corresponding to the target database cluster.

[0157] The second migration sub-unit 80533 is used to split the data table partition of the source relational database into multiple data sub-tables if the data type of the data items corresponding to the data to be migrated is Class II data, and send the multiple data sub-tables to the target database corresponding to the target database cluster in parallel, and then merge the data according to the data field mapping configuration information.

[0158] The third migration sub-unit 80534 is used to simulate the source relational database client through the data migration scheduling center if the data type of the data items corresponding to the data to be migrated is Category 3 data, parse the underlying data transmission protocol of the source relational database, read the data to be migrated and synchronize it to the target database corresponding to the target database cluster, and then merge the data according to the data field mapping configuration information.

[0159] In this embodiment, data types are divided into three categories according to the number of data items, and each category adopts a different data migration method. In particular, when the number of data items reaches three types of data, the process of splitting the data table in the source relational database into data sub-tables according to data partitions will be very slow, because it also relies on data query capabilities internally, and most relational databases have very low processing efficiency when the data volume is at the level of three types of data. At this time, relying on its processing efficiency to perform data sub-table splitting will take a long time. Therefore, adopting different data migration strategies according to different data types is effective in achieving the maximum throughput of migration.

[0160] In some optional implementations of this embodiment, the above-mentioned third migration sub-unit 80534 is also used to: based on the data field mapping configuration information, split the target data table that needs to be synchronized by the target database into multiple sub-data tables, and send them to the source relational database to obtain multiple data tables to be migrated corresponding to the multiple sub-data tables.

[0161] Call the calling interface of the data migration scheduling center to control the target database cluster to issue data transmission commands to disguise as the source relational database client to parse the underlying data transmission protocol of the source relational database, and read multiple data tables to be migrated through streaming data reading.

[0162] Merge the data of multiple data tables to be migrated.

[0163] In this embodiment, for the case where the data type belongs to the third category of data, the target data table library data table is first split and sent to the source relational database to prepare a unified data view, and then disguised as a source relational table for integration, thereby achieving rapid reading and migration of billions of data.

[0164] In some optional implementations of this embodiment, refer to Fig.12 As shown, the device 800 also includes a recording module 806 and a calculation module 807.

[0165] The recording module 806 is used to record the number of data items remaining to be migrated in the source relational database and the number of migrated data items that have been migrated to the target database.

[0166] The calculation module 807 is used to calculate the current synchronization progress according to the number of remaining data to be migrated and the number of migrated data, and select a corresponding data interface for a third party to read data according to the current synchronization progress.

[0167] In this embodiment, during the migration of the data to be migrated, the recording module 806 of the data migration scheduling center can record in real time the number of remaining data to be migrated in each data table of the source relational database and the number of migrated data that have been pulled from the target database. Then, the calculation module 807 can calculate the current synchronization progress based on the total number of data to be migrated, the number of remaining data to be migrated, and the number of migrated data, and provide a corresponding data interface in real time for any third party to read the data.

[0168] To solve the above technical problems, the present application also provides a computer device. Fig.12 , Fig.12 This is a basic structural block diagram of the computer device in this embodiment.

[0169] The computer device 130 includes a memory 1301, a processor 1302, and a network interface 1303 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 130 with components 1301-1303, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital processors (DSP), embedded devices, etc.

[0170] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0171] The memory 1301 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 1301 can be an internal storage unit of the computer device 130, such as a hard disk or memory of the computer device 130. In other embodiments, the memory 1301 can also be an external storage device of the computer device 130, such as a plug-in hard disk equipped on the computer device 130, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 1301 can also include both the internal storage unit of the computer device 130 and its external storage device. In this embodiment, the memory 1301 is generally used to store the operating system and various application software installed on the computer device 130, such as a computer-readable instruction of a data migration method, etc. In addition, the memory 1301 can also be used to temporarily store various types of data that have been output or are to be output.

[0172] In some embodiments, the processor 1302 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 1302 is generally used to control the overall operation of the computer device 130. In this embodiment, the processor 1302 is used to run computer-readable instructions stored in the memory 1301 or process data, such as computer-readable instructions for running a data migration method.

[0173] The network interface 1303 may include a wireless network interface or a wired network interface, and the network interface 1303 is generally used to establish a communication connection between the computer device 130 and other electronic devices.

[0174] In an embodiment of the present invention, by providing a data migration scheduling center, unified management of the static information of the source relational database and the target database cluster can be achieved. Secondly, when any database in the target database cluster already has a cache of data to be migrated, the data to be migrated can be directly pulled from the database with the data cache based on the data table mapping configuration information and the data field mapping configuration information of the source relational database and the target database cluster, that is, the sub-database data can be directly pulled from each other to avoid reading from the source relational database every time, which is conducive to accelerating the synchronization progress of multiple databases in the target database cluster. In addition, when the target database cluster does not have a data cache, different data migration strategies can be selected based on the data table mapping configuration information and the data field mapping configuration information of the source relational database and the target database cluster to perform data synchronization migration according to the data type of the data to be migrated, which is conducive to achieving the maximum throughput of the migration and improving the efficiency of data migration.

[0175] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of a data migration method as described above.

[0176] In an embodiment of the present invention, by providing a data migration scheduling center, unified management of the static information of the source relational database and the target database cluster can be achieved. Secondly, when any sub-database in the target database cluster already has data to be migrated, the data to be migrated can be directly pulled from the sub-database where the data to be migrated exists based on the data table mapping configuration information and the data field mapping configuration information of the source relational database and the target database cluster. That is, when the target database cluster needs to configure the cluster to have multiple sub-databases, the sub-database data can be directly pulled from each other to avoid reading from the source relational database every time, which is conducive to accelerating the synchronization progress of multiple sub-databases in the target database cluster. In addition, when it is necessary to migrate the data to be migrated from the source relational database to the target database cluster, different data migration strategies can be selected based on the data table mapping configuration information and the data field mapping configuration information of the source relational database and the target database cluster to perform data synchronization migration according to the data type to which the data to be migrated belongs, which is conducive to achieving the maximum throughput of migration and improving data migration efficiency.

[0177] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0178] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A data migration method, applicable to a data migration system of an intelligent device, the system comprising a data migration scheduling center, a source relational database and a target database cluster, characterized in that: The steps include: Establishing a connection between the data migration scheduling center and the source relational database and the target database cluster, and matching the source relational database and the target database cluster according to static information of the source relational database and the target database cluster, wherein the source relational database includes data to be migrated; Detecting whether the data cache to be migrated already exists in the target database cluster; If the target database cluster has the data cache to be migrated, then based on the existing data cache and the database connection configuration information stored in the data migration scheduling center, directly pull the data to be migrated from the database where the data to be migrated exists to the database where the data has not been migrated; If the target database cluster does not have the data cache to be migrated, detecting whether the source relational database has the data to be migrated; If the data to be migrated exists in the source relational database, based on the static information of the source relational database and the target database cluster, the corresponding data migration strategy is mobilized according to different data types to migrate the data to be migrated from the source relational database to the corresponding target database in the target database cluster, and a migrated data cache is established in the target database; The static information includes data table mapping configuration information, data field mapping configuration information and database connection configuration information. The steps of establishing a connection between the data migration scheduling center and the source relational database and the target database cluster, and matching according to the static information of the source relational database and the target database cluster include: Sending a first connection request signal to the source relational database through the data migration scheduling center, and sending a second connection request signal to the target database cluster; Establishing a network connection between the source relational database and the target database cluster according to a first verification result returned by the source relational database and a second verification result returned by the target database cluster; Matching and mapping the data tables and data fields of the source relational database and the target database cluster according to the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information; The step of mobilizing corresponding data migration strategies based on static information of the source relational database and the target database cluster according to different data types, migrating the data to be migrated from the source relational database to the corresponding target database in the target database cluster, and establishing a cache of migrated data in the target database specifically includes: Simulating a source relational database client through the data migration scheduling center and sending a data query request to the source relational database; Simulating a target database client through the data migration scheduling center and sending a data write request to the target database; According to the data query connection result returned by the source relational database, the data write connection result returned by the target database, the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information, the corresponding data migration strategy is mobilized based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster, and a migrated data cache is established in the target database.

2. A data migration method according to claim 1, characterized in that: The data types include first-class data, second-class data, and third-class data with increasing data volumes. The step of mobilizing corresponding data migration strategies based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster specifically includes: Read and determine the data types corresponding to the number of data items corresponding to the data to be migrated in each data table of the source relational database; If the data type of the number of data items corresponding to the data to be migrated is the first type of data, then based on the preset number of unit data items, the data is polled and read from the source relational database to the data migration scheduling center, and then sent to the target database corresponding to the target database cluster; If the data type of the data items corresponding to the data to be migrated is the second type of data, the data table partition of the source relational database is split into multiple data sub-tables, and the multiple data sub-tables are sent to the target database corresponding to the target database cluster in parallel, and then the data is merged according to the data field mapping configuration information; If the data type of the number of data items corresponding to the data to be migrated is the three types of data, the data migration scheduling center simulates the source relational database client, parses the underlying data transmission protocol of the source relational database, reads the data to be migrated and synchronizes it to the target database corresponding to the target database cluster, and then merges the data according to the data field mapping configuration information.

3. A data migration method according to claim 2, characterized in that: The step of simulating the source relational database client by the data migration scheduling center, parsing the underlying data transmission protocol of the source relational database, reading the data to be migrated and synchronizing it to the target database corresponding to the target database cluster for data merging specifically includes: Based on the data field mapping configuration information, the target data table to be synchronized by the target database is split into multiple sub-data tables, and the sub-data tables are sent to the source relational database to obtain multiple data tables to be migrated corresponding to the multiple sub-data tables; Calling the calling interface of the data migration scheduling center, controlling the target database to issue a data transmission command to disguise as the source relational database client to parse the underlying data transmission protocol of the source relational database, and reading the multiple data tables to be migrated by streaming data reading; The data of the multiple data tables to be migrated that are read are merged.

4. A data migration method according to claim 1, characterized in that: After the step of mobilizing the corresponding data migration strategy according to different data types based on the static information of the source relational database and the target database cluster to migrate the data to be migrated from the source relational database to the target database corresponding to the target database cluster, the method further includes: Record the number of data items remaining to be migrated in the source relational database and the number of migrated data items that have been migrated to the target database; The current synchronization progress is calculated according to the number of remaining data items to be migrated and the number of migrated data items, and a corresponding data interface is selected according to the current synchronization progress for a third party to read data.

5. A data migration device, applicable to a data migration system of an intelligent device, the system comprising a data migration scheduling center, a source relational database and a target database cluster, characterized in that: include: A connection module, used to establish a connection between the data migration scheduling center and the source relational database and the target database cluster, and to match the source relational database and the target database cluster according to static information of the source relational database and the target database cluster, wherein the source relational database includes data to be migrated; A first detection module, used to detect whether the data cache to be migrated already exists in the target database cluster; A first scheduling module is used for directly pulling the data to be migrated from the database where the data to be migrated exists to the database where the data has not been migrated, based on the existing data cache and the database connection configuration information stored in the data migration scheduling center, if the target database cluster has the data cache to be migrated; A second detection module, configured to detect whether the source relational database has the data to be migrated if the target database cluster does not have the data cache to be migrated; A second scheduling module is used for, if the data to be migrated exists in the source relational database, mobilizing corresponding data migration strategies according to different data types based on the static information of the source relational database and the target database cluster to migrate the data to be migrated from the source relational database to the corresponding target database in the target database cluster, and establishing a migrated data cache in the target database; The static information includes data table mapping configuration information, data field mapping configuration information and database connection configuration information, and is characterized in that the connection module includes: A first request submodule, configured to send a first request connection signal to the source relational database through the data migration scheduling center, and send a second request connection signal to the target database cluster; A connection submodule, configured to establish a network connection between the source relational database and the target database cluster according to a first verification result returned by the source relational database and a second verification result returned by the target database cluster; A mapping submodule, used to match and map the data tables and data fields of the source relational database and the target database cluster according to the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information; The second scheduling module includes: a second request submodule, a third request submodule and a scheduling submodule, wherein: The second request submodule is used to simulate the source relational database client through the data migration scheduling center and send a data query request to the source relational database; The third request submodule is used to simulate the target database client through the data migration scheduling center and send a data write request to the target database; The scheduling submodule is used to query the connection results returned by the source relational database, write the connection results returned by the target database, the data table mapping configuration information, the data field mapping configuration information and the database connection configuration information, and mobilize the corresponding data migration strategy based on different data types to migrate the data to be migrated to the target database corresponding to the target database cluster, and establish a cache of migrated data in the target database.

6. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the data migration method according to any one of claims 1 to 4 when executing the computer-readable instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the data migration method according to any one of claims 1 to 4 are implemented.

Images