Data migration method, device, apparatus, and storage medium

Abstract

The present disclosure provides a data migration method, device and equipment and storage medium, relates to cloud computing, cloud storage, cloud network and cloud database technology, and can be applied to an intelligent cloud scene. A specific implementation scheme is as follows: in response to receiving a data migration request, obtaining a to-be-migrated data table corresponding to the data migration request, wherein the to-be-migrated data table comprises data information of at least one to-be-migrated data set, the to-be-migrated data set comprises first data stored in a first database and second data stored in a second database; for each data information in the to-be-migrated data table, obtaining the first data corresponding to each data information from the first database, and writing the first data into a message queue; and for each first data in the message queue, obtaining the second data corresponding to each first data from the second database, and writing each first data and the second data corresponding to each first data into a target database.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to cloud computing, cloud storage, cloud network and cloud database technology, which can be applied in an intelligent cloud scenario. BACKGROUND

[0002] A database is a warehouse for organizing, storing and managing data according to a data structure. With the development of Internet technology, the database will be updated and replaced, at which time the data in the old database needs to be migrated to the new database. Artificial participation in database migration is tedious and inefficient. SUMMARY

[0003] The present disclosure provides a data migration method, device, equipment, storage medium and program product.

[0004] According to an aspect of the present disclosure, a data migration method is provided, comprising: in response to receiving a data migration request, obtaining a to-be-migrated data table corresponding to the data migration request, wherein the to-be-migrated data table comprises data information of at least one to-be-migrated data set, and the to-be-migrated data set comprises first data stored in a first database and second data stored in a second database; for each data information in the to-be-migrated data table, obtaining first data corresponding to the each data information from the first database, and writing the first data into a message queue; and for each first data in the message queue, obtaining second data corresponding to the each first data from the second database, and writing the each first data and the second data corresponding to the each first data into a target database.

[0005] According to another aspect of the present disclosure, a data migration device is provided, comprising: an obtaining module configured to, in response to receiving a data migration request, obtain a to-be-migrated data table corresponding to the data migration request, wherein the to-be-migrated data table comprises data information of at least one to-be-migrated data set, and the to-be-migrated data set comprises first data stored in a first database and second data stored in a second database; a first writing module configured to, for each data information in the to-be-migrated data table, obtain first data corresponding to the each data information from the first database, and write the first data into a message queue; and a second writing module configured to, for each first data in the message queue, obtain second data corresponding to the each first data from the second database, and write the each first data and the second data corresponding to the each first data into a target database.

[0006] Another aspect of the present disclosure provides an electronic device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method shown in the embodiments of the present disclosure.

[0007] According to another aspect of the embodiments of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to make the computer perform the method shown in the embodiments of the present disclosure.

[0008] According to another aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer programs / instructions, characterized by that the computer programs / instructions are executed by a processor to implement the steps of the method shown in the embodiments of the present disclosure.

[0009] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0011] Figure 1 is a schematic diagram of a system architecture according to an embodiment of the present disclosure;

[0012] Figure 2 The flowchart schematically shows a data migration method according to an embodiment of the present disclosure;

[0013] Figure 3 The flowchart schematically shows a data migration method according to another embodiment of the present disclosure;

[0014] Figure 4 The flowchart schematically shows a data migration method according to another embodiment of the present disclosure;

[0015] Figure 5 The schematic diagram schematically shows a migration state according to an embodiment of the present disclosure;

[0016] Figure 6 The schematic diagram schematically shows a data migration according to an embodiment of the present disclosure;

[0017] Figure 7 The block diagram schematically shows a data migration device according to an embodiment of the present disclosure; and

[0018] Figure 8A block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding them. These should be considered in a descriptive sense only and not limiting. Therefore, it will be recognized by those of ordinary skill that various changes and modifications can be made to the embodiments described and illustrated herein without departing from the scope and spirit of the present disclosure. Also, in the interest of clarity, not all of the routine features of the implementations described herein are shown and described. Further, it will be appreciated that the apparatus and system embodiments can be implemented by computer program instructions organized into one or more software modules in the memory of the device, which when executed by the processor of the device implement the methods described herein.

[0020] The following will be described in conjunction with Figure 1 A system architecture to which the data migration method and device provided by the present disclosure can be applied is described.

[0021] Figure 1 is a schematic diagram of a system architecture according to an embodiment of the present disclosure. It should be noted that, Figure 1 The system architecture shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0022] As Figure 1 shown, the system architecture 100 can include source databases 101, 102, a data transmission system 103 and a target database 104.

[0023] According to an embodiment of the present disclosure, the data transmission system 103 can provide, for example, data migration, data synchronization, data subscription and other database data transmission services (Data Transmission Service, DTS).

[0024] According to an embodiment of the present disclosure, the source database 101, the source database 102 and the target database 104 can be different types of databases, or can be the same type of database, which is not specifically limited by the present disclosure. The types include, but are not limited to, MySQL, mola and TafDB.

[0025] According to an embodiment of the present disclosure, any of the source database 101, the source database 102 and the target database 104 can be located in the same database system, and the source database 101, the source database 102 and the target database 104 can also be located in different database systems, which is not specifically limited by the present disclosure.

[0026] According to an embodiment of the present disclosure, when data is stored, the data can be divided into two parts in any proportion, one part is stored in the source database 101 and the other part is stored in the source database 102.

[0027] When data in the source databases 101 and 102 needs to be migrated to the target database 104, the data transmission system 103 can obtain the data to be migrated from the source databases 101 and 102, and then write the data to be migrated to the target database 104.

[0028] The data transmission system 103 can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in a cloud computing service system to solve the defects of great management difficulty and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server, or VPS for short). The data transmission system 103 can also be a server of a distributed system, or a server combined with a blockchain.

[0029] It should be understood that Figure 1 The number of source databases, target databases and servers in the above description is only illustrative. According to the needs of implementation, there can be any number of source databases, target databases and servers.

[0030] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0031] In the technical solutions of the present disclosure, the authorization or consent of the user is obtained before the user's personal information is obtained or collected.

[0032] The data migration method provided by the present disclosure will be described below. Figure 2 The data migration method provided by the present disclosure will be described below.

[0033] Figure 2 A flowchart of a data migration method according to an embodiment of the present disclosure is schematically shown. The data migration method can be performed by the data transmission system shown above, for example.

[0034] As shown in Figure 2 The data migration method 200 includes, at operation S210, in response to receiving a data migration request, obtaining a data table to be migrated corresponding to the data migration request.

[0035] According to an embodiment of the present disclosure, a data migration request can be used to request a data transmission system to migrate to-be-migrated data in a source database to a target database. The to-be-migrated data table includes data information of at least one to-be-migrated data set. The to-be-migrated data set includes first data stored in a first database and second data stored in a second database. The data information of the to-be-migrated data set may, for example, include an identifier of the to-be-migrated data set. The to-be-migrated data set may, for example, be of a bucket type. The identifier of the to-be-migrated data set may, for example, be an ID of the bucket.

[0036] According to an embodiment of the present disclosure, the data information of the to-be-migrated data set can be written into the to-be-migrated data table in advance. Each to-be-migrated data set may, for example, include a plurality of data, such as metadata. Metadata may, for example, be information used to describe properties of data, and is commonly used to support functions such as indicating storage locations of data, searching for historical data or resources, recording files, and the like. The data in the to-be-migrated data set may, for example, be stored in the first database and the second database in any proportion.

[0037] According to an embodiment of the present disclosure, the to-be-migrated data table may, for example, be stored in the source database, for example, in the first database and / or the second database. In an example, the to-be-migrated data table may, for example, be stored in the first database. The data transmission system may, for example, obtain the to-be-migrated data table from the first database.

[0038] Then, in operation S220, for each data information in the to-be-migrated data table, the first database is queried for first data corresponding to each data information, and the first data is written into a message queue.

[0039] According to an embodiment of the present disclosure, the data matching the data information may, for example, be searched for in the first database as the first data.

[0040] According to an embodiment of the present disclosure, the first data corresponding to each data information may, for example, be stored in one message queue, or may, for example, be stored in a plurality of message queues, without limitation.

[0041] In operation S230, for each first data in the message queue, the second database is queried for second data corresponding to each first data, and each first data and the second data corresponding to each first data are written into the target database.

[0042] According to an embodiment of the present disclosure, a first data can be obtained from the message queue, for example, according to a predetermined order. The predetermined order may, for example, include a time order in which the first data is written into the message queue, and the first data written into the message queue first is migrated first. Data information corresponding to the first data is determined. Then, data matching the data information is searched for in the second database as second data corresponding to the first data. The first data and the second data are stored together into the target database. Next, the next first data can be obtained from the message queue according to the predetermined order, and the above operation is repeated until all data in the message queue is migrated.

[0043] The related art involves manual participation in database migration, which is complicated and inefficient.

[0044] According to the data migration method of the disclosed embodiment, the data in the source database can be automatically migrated to the target database, and the data migration is efficient. In addition, by using the message queue for data migration, the order of the data after migration can be ensured.

[0045] According to another embodiment of the present disclosure, the data transmission system may, for example, include a plurality of first nodes. The data transmission system may, for example, determine a starting node from the plurality of first nodes according to a consensus algorithm. The consensus algorithm may, for example, include a raft algorithm. Based on this, in the present embodiment, the starting node can be used to obtain first data corresponding to each data information from the first database and write the first data into the message queue.

[0046] According to another embodiment of the present disclosure, the data transmission system may, for example, include a plurality of second nodes. The data transmission system may, for example, determine a data migration node from the plurality of first nodes according to a consensus algorithm. The data migration node can serve as a master, and at least one worker node can be configured for the data migration node. Based on this, in the present embodiment, the data migration node can be used to generate at least one migration task according to each first data in the message queue and assign the at least one migration task to at least one worker node corresponding to the data migration node. Then, the at least one worker node can be used to obtain second data corresponding to each first data from the second database according to the assigned migration task, and write each first data and each second data into the target database.

[0047] According to another embodiment of the present disclosure, the water level information in the data migration node may, for example, also be statistical water level information. The water level information is used to represent the position of the last reading in the message queue. If there are multiple message queues, the minimum position is selected from the positions of the last reading in the message queues as the water level position. The starting node may, for example, poll the data migration node to determine the water level information in the data migration node. In the case where the water level information meets a predetermined condition, the read-write request for the data set to be migrated is forwarded to the target database. The predetermined condition may, for example, include that the sending time of the data corresponding to the water level information is consistent with the current time of the system. If the water level information meets the predetermined condition, it means that the data is complete for migration. If the water level information does not meet the predetermined condition, it means that the data is not complete for migration.

[0048] Figure 3 A flowchart of a data migration method according to another embodiment of the present disclosure is schematically shown. The data migration method may, for example, be performed by the data transmission system shown above.

[0049] As shown in the above, Figure 3 In the case where each first data and the second data corresponding to each first data are written into the target database, the data migration method 300 may further include, at operation S340, checking whether the data written in the target database is consistent with the data set to be migrated in the first database and the second database.

[0050] In the case where the data written in the target database is not consistent with the data set to be migrated in the first database and the second database, the data written in the target database is deleted at operation S350.

[0051] According to an embodiment of the present disclosure, if the data written in the target database is consistent with the data set to be migrated in the first database and the second database, it means that the data migration is successful. If the data written in the target database is not consistent with the data set to be migrated in the first database and the second database, it means that the data migration fails, and in this case, the data written in the target database can be deleted to avoid conflicts in the data in the target database when the data migration is reinitiated.

[0052] Figure 4 A flowchart of a data migration method according to another embodiment of the present disclosure is schematically shown. The data migration method may, for example, be performed by the data transmission system shown above.

[0053] As shown in the above, Figure 4 The data migration method 400 may further include, at operation S460, obtaining the database change log of the first database.

[0054] According to an embodiment of the present disclosure, the database change log can be used to record logs of all database table structure changes and table data modifications in the first database. The database change log may, for example, include a binlog log.

[0055] In operation S470, according to the database change log, the incremental data changed in the first database within a predetermined time after receiving the data migration request is determined.

[0056] According to an embodiment of the present disclosure, the predetermined time can be set according to actual needs, for example, the average migration time can be calculated according to the time length consumed by each historical data migration as the predetermined time.

[0057] According to an embodiment of the present disclosure, for example, according to the database change log, the data in which the database table structure is changed and / or the table data is modified within the predetermined time can be determined as the incremental data.

[0058] In operation S480, the incremental data is written to the message queue as the first data.

[0059] According to an embodiment of the present disclosure, by writing the incremental data in the first database to the message queue, the incremental data in the first database and the corresponding incremental data in the second database can be migrated to the target database subsequently, thereby ensuring the consistency of the data before and after migration.

[0060] According to another embodiment of the present disclosure, the start node and the data migration node can also record the migration state. Based on this, Figure 5 A schematic diagram of the migration state according to an embodiment of the present disclosure is schematically shown.

[0061] In Figure 5 It is shown that the migration state may, for example, include states such as "initial", "in migration", "migration complete", "check complete", "end", "delete" and "failure".

[0062] Exemplarily, in the embodiment, before the data migration node generates at least one migration task according to each first data in the message queue, the system can be in an "initial" state. In the data migration node, when at least one migration task is generated according to each first data in the message queue and each first data and each second data are written into the target database, the system can be in a "migration in progress" state. After each first data and each second data are written into the target database, the system can be in a "migration completed" state. Then data checking is performed, and if the checking is passed, the system can be transferred from the "migration completed" state to a "checking completed" state. Then the "checking completed" state can be transferred to an "end" state. When a migration task restart or migration failure occurs during the data migration process, the data written into the target database can be deleted, and at this time the system can be in a "deletion" state. After the deletion is completed, the system can be transferred from the "deletion" state to a "failure" state.

[0063] According to the embodiment of the present disclosure, the migration state is recorded in the start node and the data migration node. When at least one of the start node and the data migration node needs to be replaced, the replaced node can continue the subsequent migration work according to the recorded migration state, so that the migration task is not lost, and the reliability of the data migration is effectively improved.

[0064] The data migration method shown in the above will be further described below with reference to specific embodiments. Figure 6 Those skilled in the art can understand that the following example embodiments are only used to understand the present disclosure, and the present disclosure is not limited thereto.

[0065] Figure 6 A schematic diagram of data migration according to an embodiment of the present disclosure is schematically shown.

[0066] In Figure 6 It is shown that the data to be migrated can be metadata, for example. The metadata can be stored in the form of bucket data, that is, the data set to be migrated can be of the bucket type, for example.

[0067] The source database can initiate a migration request to the data processing system. The source database can include a first source database and a second source database. Exemplarily, in the embodiment, the first source database can include a MySQL database, for example. The second source database can include a mola database, for example. The target database can include a TafDB database, for example.

[0068] The start node of the data processing system can include a DTS-start node, for example. The start node can be a central service composed of a raft protocol, and the start node can maintain a list of available message queues (pipelets) in the memory.

[0069] The DTS-start node may, for example, obtain data in the data table to be migrated, mark the data as being in migration. Then record the message queue information corresponding to the bucket, the starting subscription point, the data information into the source database, and additionally record the migration start time, and then call the API (application program interface) of the DTS service to initiate the migration task of the stock data and the incremental data. The DTS service may obtain the first data corresponding to each data information from the first database, and write the first data into the message queue.

[0070] The data migration node of the data processing system may include a master node. The master node may generate at least one migration task according to each first data in the message queue, and assign the at least one migration task to at least one worker node corresponding to the data migration node. Each worker node requests the data migration node to obtain the assigned migration task, and then executes the assigned migration task, that is, obtains the second data corresponding to each first data from the second database, and writes each first data and each second data into the target database. In this way, the data to be migrated in the first source database and the second source database can be migrated to the target database.

[0071] The following will be described in conjunction with Figure 7 The data migration apparatus provided by the present disclosure is described.

[0072] Figure 7 A block diagram of a data migration apparatus according to an embodiment of the present disclosure is schematically shown.

[0073] As Figure 7 shown, the data migration apparatus 700 includes an obtaining module 710, a first writing module 720, and a second writing module 730. The data migration apparatus 700 can be applied to a data transmission system.

[0074] The obtaining module 710 is configured to, in response to receiving a data migration request, obtain a data table to be migrated corresponding to the data migration request, wherein the data table to be migrated includes data information of at least one data set to be migrated, and the data set to be migrated includes first data stored in a first database and second data stored in a second database.

[0075] The first writing module 720 is configured to, for each data information in the data table to be migrated, obtain first data corresponding to each data information from the first database, and write the first data into a message queue.

[0076] The second writing module 730 is configured to, for each first data in the message queue, obtain second data corresponding to each first data from the second database, and write each first data and the second data corresponding to each first data into a target database.

[0077] According to an embodiment of the present disclosure, the data transmission system can comprise a plurality of first nodes, and the apparatus can further comprise a first determining module configured to determine a starting node from the plurality of first nodes according to a consensus algorithm. The first writing module can be further configured to obtain first data corresponding to each data information from the first database by using the starting node, and write the first data into the message queue.

[0078] According to an embodiment of the present disclosure, the data transmission system can comprise a plurality of second nodes, and the apparatus can further comprise a second determining module configured to determine a data migration node from the plurality of first nodes according to a consensus algorithm, and configure at least one worker node for the data migration node. The second writing module can be further configured to generate at least one migration task according to each first data in the message queue by using the data migration node, and assign the at least one migration task to at least one worker node corresponding to the data migration node. And obtain second data corresponding to each first data from the second database according to the assigned migration task by using the at least one worker node, and write each first data and each second data into the target database.

[0079] According to an embodiment of the present disclosure, the apparatus can further comprise a checking module and a deleting module. The checking module can be configured to check whether the data written in the target database is consistent with the to-be-migrated data set in the first database and the second database in the case of writing each first data and the second data corresponding to each first data into the target database. The deleting module can be configured to delete the data already written in the target database in the case that the data written in the target database is not consistent with the to-be-migrated data set in the first database and the second database.

[0080] According to an embodiment of the present disclosure, the apparatus can further comprise a log obtaining module, an incremental data determining module, and an incremental data writing module. The log obtaining module can be configured to obtain a database change log of the first database. The incremental data determining module can be configured to determine incremental data changed in the first database within a predetermined time after receiving the data migration request according to the database change log. The incremental data writing module can be configured to write the incremental data as the first data into the message queue.

[0081] According to an embodiment of the present disclosure, the apparatus can further comprise a water level determining module and a forwarding module. The water level determining module can be configured to determine water level information in the data migration node. The forwarding module can be configured to forward a read-write request for the to-be-migrated data set to the target database in the case that the water level information meets a predetermined condition.

[0082] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0083] Figure 8 A block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown schematically. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0084] As shown in Figure 8 The electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0085] Various components in the electronic device 800 are connected to the I / O interface 805, including an input unit 806, such as a keyboard, a mouse, and the like; an output unit 807, such as various types of displays, speakers, and the like; the storage unit 808, such as a magnetic disk, an optical disk, and the like; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0086] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the data migration method. For example, in some embodiments, the data migration method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the data migration method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the data migration method by any other appropriate means, such as by means of firmware.

[0087] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0088] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0089] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0091] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0092] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0093] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0094] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data migration method, comprising: In response to receiving a data migration request, a data table to be migrated corresponding to the data migration request is obtained, wherein the data table to be migrated includes data information of at least one dataset to be migrated, and the dataset to be migrated includes first data stored in a first database and second data stored in a second database; For each piece of data in the data table to be migrated, retrieve the first data corresponding to each piece of data from the first database and write the first data into a message queue; and For each first data in the message queue, retrieve the second data corresponding to each first data from the second database, and write each first data and the second data corresponding to each first data into the target database; The method is applied to a data transmission system, wherein the data transmission system includes a plurality of first nodes; the method further includes: determining a data migration node from the plurality of first nodes according to a consensus algorithm, and configuring at least one worker node for the data migration node; The step of retrieving the second data corresponding to each first data in the message queue from the second database and writing each first data and the second data corresponding to each first data into the target database includes: using the data migration node to generate at least one migration task based on each first data in the message queue, and assigning the at least one migration task to at least one worker node corresponding to the data migration node; and using the at least one worker node to retrieve the second data corresponding to each first data from the second database according to the assigned migration task, and writing each first data and each second data into the target database.

2. The method according to claim 1, wherein, The method further includes: The starting node is determined from the plurality of first nodes according to the consensus algorithm; Specifically, the step of retrieving first data corresponding to each data piece of information from the first database and writing the first data to a message queue for each data piece of information in the data table to be migrated includes: Using the startup node, first data corresponding to each piece of data information is obtained from the first database, and the first data is written into the message queue.

3. The method according to claim 1, further comprising: When writing each first data and the second data corresponding to each first data into the target database, check whether the data written into the target database is consistent with the dataset to be migrated in the first database and the second database; and If the data written to the target database is inconsistent with the dataset to be migrated in the first and second databases, the data already written to the target database shall be deleted.

4. The method according to claim 1, further comprising: Retrieve the database change log for the first database; Based on the database change log, determine the incremental data that has changed in the first database within a predetermined time after receiving the data migration request; as well as The incremental data is written into the message queue as the first data.

5. The method according to claim 2, further comprising: Determine the water level information in the data migration node; as well as If the water level information meets the predetermined conditions, the read and write requests for the dataset to be migrated will be forwarded to the target database.

6. A data migration apparatus, comprising: The acquisition module is used to acquire a data table to be migrated corresponding to the data migration request in response to receiving the data migration request. The data table to be migrated includes data information of at least one dataset to be migrated. The dataset to be migrated includes first data stored in a first database and second data stored in a second database. The first writing module is configured to, for each piece of data in the data table to be migrated, retrieve first data corresponding to each piece of data from the first database, and write the first data into a message queue; and The second writing module is used to retrieve the second data corresponding to each first data from the second database for each first data in the message queue, and write each first data and the second data corresponding to each first data into the target database; The device is applied to a data transmission system, wherein the data transmission system includes a plurality of first nodes; the device further includes: a second determining module, configured to determine a data migration node from the plurality of first nodes according to a consensus algorithm, and configure at least one worker node for the data migration node; The second writing module is further configured to: generate at least one migration task based on each first data in the message queue using the data migration node, and assign the at least one migration task to at least one worker node corresponding to the data migration node; and use the at least one worker node to obtain second data corresponding to each first data from the second database according to the assigned migration task, and write each first data and each second data into the target database.

7. The apparatus according to claim 6, wherein, The device further includes: The first determining module is used to determine the starting node from the plurality of first nodes according to the consensus algorithm; The first writing module is further configured to: Using the startup node, first data corresponding to each piece of data information is obtained from the first database, and the first data is written into the message queue.

8. The apparatus according to claim 6, further comprising: The inspection module is used to check whether the data written to the target database is consistent with the dataset to be migrated in the first database and the second database when each first data and the second data corresponding to each first data are written to the target database. as well as The deletion module is used to delete data already written to the target database if the data written to the target database is inconsistent with the dataset to be migrated in the first database and the second database.

9. The apparatus according to claim 6, further comprising: The log acquisition module is used to acquire the database change logs of the first database; The incremental data determination module is used to determine, based on the database change log, the incremental data that has changed in the first database within a predetermined time after receiving the data migration request; as well as An incremental data writing module is used to write the incremental data as the first data into the message queue.

10. The apparatus according to claim 7, further comprising: A water level determination module is used to determine the water level information in the data migration node; as well as The forwarding module is used to forward read and write requests for the dataset to be migrated to the target database when the water level information meets predetermined conditions.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

13. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-5.

Images