Full data migration method, device and equipment and computer readable storage medium

Abstract

The application provides a full-amount data migration method, device and equipment and a computer readable storage medium, applied to the field of cloud technology, and the method comprises the following steps: acquiring full-amount data in a source database; the full-amount data comprises at least one migration object; migration objects are extracted from the at least one migration object of the full-amount data in sequence as to-be-migrated objects, and the to-be-migrated objects are migrated from the source database to a target database through an idle migration process in at least one migration process; and in the case that the at least one migration object is all migrated to the target database, a full-amount data migration process is completed. Through the full-amount data migration method provided by the application, the migration efficiency of the full-amount data can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a data migration technology, in particular to a full data migration method and device, equipment and computer readable storage medium. BACKGROUND

[0002] A database is a warehouse for organizing, storing and managing data according to a data structure. With the development of Internet technology, in the scenarios of upgrading the hardware of a database server, the data of the database often needs to be migrated from one electronic device to another electronic device.

[0003] In the related art, in the process of transmitting full data in a source database to a target database, the full data is exported to an external storage pool, and after the full data is exported, the full data in the storage pool is imported into the target database. As can be seen, in the related art, there are full export and full import processes for full data, and full export must be completed before full import can be started, which has a global synchronization barrier and seriously affects the migration efficiency of full data. SUMMARY

[0004] The embodiments of the present application provide a full data migration method, device, equipment and computer readable storage medium, which can improve the migration efficiency of full data.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide a full data migration method, which comprises: a full data migration method, characterized by comprising: obtaining full data in a source database; the full data comprises at least one migration object; sequentially extracting a migration object from at least one migration object of the full data as a to-be-migrated object, and migrating the to-be-migrated object from the source database to a target database through an idle migration process in at least one migration process; and in the case that all the at least one migration object is migrated to the target database, completing the full data migration process.

[0007] In some embodiments of the present application, the sequentially extracting a migration object from at least one migration object of the full data as a to-be-migrated object, and migrating the to-be-migrated object from the source database to a target database through an idle migration process in at least one migration process comprises: in the case that there is an idle migration process in at least one migration process, extracting a migration object from at least one migration object as a to-be-migrated object; assigning the to-be-migrated object to the idle migration process; and migrating the to-be-migrated object from the source database to the target database through the idle migration process.

[0008] In some embodiments of the present application, the migrating the to-be-migrated object from the source database to the target database by the idle migration process comprises: taking the to-be-migrated object from the source database by a reading thread in the idle migration process, and writing the to-be-migrated object into a data queue corresponding to the idle migration process in the memory; taking the to-be-migrated object from the data queue by a writing thread in the idle migration process, and writing the to-be-migrated object into the target database.

[0009] In some embodiments of the present application, the taking the to-be-migrated object from the source database by the reading thread in the idle migration process, and writing the to-be-migrated object into a data queue corresponding to the idle migration process in the memory comprises: reading structural data of the to-be-migrated object from the source database to the data queue by the reading thread; reading source database data of the to-be-migrated object from the source database to the data queue by the reading process.

[0010] In some embodiments of the present application, the taking the to-be-migrated object from the data queue by the writing thread in the idle migration process, and writing the to-be-migrated object into the target database comprises: taking the structural data of the to-be-migrated object from the data queue by the writing thread, and writing the structural data into the target database to complete a configuration process of the target database according to the structural data; taking the source database data of the to-be-migrated object from the data queue by the writing thread, and writing the source database data into the target database configured by the structural data.

[0011] In some embodiments of the present application, the method further comprises: monitoring a process state of each migration process in the at least one migration process; the process state comprises a reading thread state and a writing thread state; determining the idle migration process in the at least one migration process according to the reading thread state and the writing thread state corresponding to each migration process.

[0012] In some embodiments of the present application, the determining the idle migration process in the at least one migration process according to the reading thread state and the writing thread state corresponding to each migration process comprises at least one of the following: in a case where the reading thread state and the writing thread state corresponding to the migration process are all idle states, determining the migration process as the idle migration process; in a case where the reading thread state corresponding to the migration process is an idle state, and the writing thread state is all working states, determining the migration process as the idle migration process.

[0013] In some embodiments of the present application, the method further comprises: in the process of taking the to-be-migrated object from the source database by the reading thread in the idle migration process and writing the to-be-migrated object into the data queue corresponding to the idle migration process in the memory, if the data queue is full, suspending the reading thread; in the process of taking the to-be-migrated object from the data queue by the writing thread in the idle migration process and writing the to-be-migrated object into the target database, obtaining the load quantization value in the data queue, and in the case that the load quantization value is lower than a preset threshold, resuming the reading thread.

[0014] The embodiment of the present application provides a full-amount data migration device, the device comprises: an acquisition module, used for acquiring full-amount data in a source database; the full-amount data comprises at least one migration object; a migration module, used for sequentially extracting a migration object from the at least one migration object of the full-amount data as a to-be-migrated object, and migrating the to-be-migrated object from the source database to a target database through an idle migration process in at least one migration process; the migration module is further used for completing a full-amount data migration process in the case that the at least one migration object is all migrated to the target database.

[0015] The embodiment of the present application provides a full-amount data migration device, comprising:

[0016] A memory, used for storing executable instructions;

[0017] A processor, used for executing the executable instructions stored in the memory, so as to realize the full-amount data migration method provided by the embodiment of the present application.

[0018] The embodiment of the present application provides a computer readable storage medium, which stores executable instructions, and is used for causing a processor to execute, so as to realize the full-amount data migration method provided by the embodiment of the present application.

[0019] The embodiment of the present application has the following beneficial effects:

[0020] The embodiment of the present application refines the full-amount data into a plurality of migration objects, and compared with the scheme that the full-amount data is first read into a read-write storage pool and then written into a target database after the reading process is completed in the prior art, the migration objects can be directly migrated into the target database respectively, and the read-write storage pool is saved, so that the migration speed is improved, the storage cost is reduced, the operation cost is reduced, the component dependence is reduced, and the overall system reliability is improved. In addition, the plurality of migration processes are used to perform parallel migration on the migration objects, so that the technical effect of simultaneously performing full-amount export and full-amount import on the full-amount data is achieved, and the overall migration efficiency of the full-amount data is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1is a schematic diagram of an optional architecture of a full-data migration system provided by an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a structure of a full-data migration device provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional flow of a full-data migration method provided by an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an optional flow of a full-data migration method provided by an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of an optional flow of a full-data migration method provided by an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of an optional flow of a full-data migration method provided by an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of an optional flow of a full-data migration method provided by an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of an optional migration process of a full-data migration method provided by an embodiment of the present application;

[0029] Figure 9 is a schematic diagram of an optional interactive interface of a database migration service provided by an embodiment of the present application;

[0030] Figure 10 is a schematic diagram of an optional secondary menu of an interactive interface provided by an embodiment of the present application;

[0031] Figure 11 is a schematic diagram of an optional migration process of a full-data migration method provided by an embodiment of the present application;

[0032] Figure 12 is a schematic diagram of an optional migration process of a full-data migration method provided by an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without making any creative labor fall within the scope of protection of the present application.

[0034] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" can be the same subset or a different subset of all possible embodiments, and can be combined with each other, without conflict.

[0035] In the following description, the terms "first\second\third" are merely to distinguish similar objects, and do not represent a specific order of the objects. It is understood that the "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments described in the present embodiment can be implemented in an order other than that illustrated or described in the present embodiment.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application is for the purpose of describing the embodiments of the present application only, and is not intended to limit the present application.

[0037] First, the terms involved in the embodiments of the present application are briefly introduced:

[0038] (1) Database (DB): In short, it can be regarded as an electronic file cabinet, that is, a place for storing electronic files. Users can add, query, update, delete, etc. The data in the file. The so-called "database" is a collection of data stored together in a certain way, shared by multiple users, with as little redundancy as possible, and independent of application programs.

[0039] (2) DTS: Data Transmission Service, also known as data migration service, is a service that supports data synchronization and transmission between multiple homogeneous or heterogeneous databases.

[0040] (3) Barrier: a synchronization method in parallel computing. For a group of processes or threads, a synchronization barrier in a program means that any thread / process execution must wait until all threads / processes reach this point before continuing to execute the following text.

[0041] (4) Full export: all existing structure data and data of the specified objects to be migrated are exported from the source database.

[0042] (5) Full import: the information exported by full export is imported into the target database.

[0043] (6) Incremental export: during the database data migration process, the source DB still has data written, which is incremental data, and the incremental data needs to be exported.

[0044] (7) Incremental import: import the exported incremental data into the target DB.

[0045] Referring to Figure 1 , Figure 1 is an optional architecture diagram of a full data migration system 100 provided by the embodiments of the present application, for implementing a full data migration application, a terminal (exemplarily shown as terminal 400-1 and terminal 400-2) connects a server 200 through a network 300, the network 300 can be a wide area network or a local area network, or a combination of the two. Figure 1 It is also shown that the server 200 can be a server cluster, which includes servers 200-1 to 200-3, similarly, the servers 200-1 to 200-3 can be physical machines, or virtual machines constructed by using virtualization technologies (such as container technology and virtual machine technology, etc.), the embodiments of the present application do not limit this, of course, a single server can also be used to provide services in the embodiments. The server 200 is also connected to a database 600, which can be a database cluster, including databases 600-1 and 600-2 in the database cluster.

[0046] In some embodiments of the present application, after receiving a full data migration request of a user, the terminal determines the source database identifier and the target database identifier in the full data migration request, and sends the source database identifier and the target database identifier to the server 200 through the network 300, the server 200 determines the source database 600-1 corresponding to the source database identifier and the target database 600-2 corresponding to the target database identifier in the database 600 according to the source database identifier and the target database identifier. The server 200 can obtain full data in the source database 600-1; the full data includes at least two migration objects; sequentially extract migration objects as to-be-migrated objects from the at least two migration objects of the full data, and migrate the to-be-migrated objects from the source database 600-1 to the target database 600-2 through an idle migration process in at least one migration process; in the case that the at least two migration objects are all migrated to the target database 600-2, the full data migration process is completed. During the migration process, the migration progress can be sent to the terminal. The terminal can display the migration progress of the full data on a graphical interface (exemplarily shown as graphical interface 410-1 and graphical interface 410-2).

[0047] The embodiments of the present application can be implemented by means of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network to realize data calculation, storage, processing, and sharing in a wide area network or a local area network.

[0048] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business model application, which can form a resource pool and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of computing and storage resources, for example, the portal website of the education system.

[0049] As an example, the server 200 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, etc. The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, and a smart watch, but is not limited thereto. The terminal and the server 200 can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application.

[0050] Referring to Figure 2 , Figure 2 is a structural schematic diagram of the full data migration device 500 provided by the embodiments of the present application, Figure 2 The full data migration device 500 shown in the full data migration device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the full data migration device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection communication between the components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540 in Figure 2 .

[0051] The processor 510 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0052] The user interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, a mouse, a microphone, a touch screen display, a camera, other input buttons and controls.

[0053] The memory 550 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (ROM), and the volatile memory can be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory. The memory 550 can optionally include one or more storage devices that are physically separate from the processor 510.

[0054] In some embodiments of the present application, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0055] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks.

[0056] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), etc.

[0057] The display module 553 is used to enable the presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., a display screen, a speaker, etc.), such as a user interface for operating peripheral devices and displaying content and information.

[0058] The input processing module 554 is used to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0059] In some embodiments of the present application, the full data migration device provided by the embodiments of the present application can be realized in a combination of software and hardware, for example, the full data migration device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the full data migration method provided by the embodiments of the present application.

[0060] In some embodiments of the present application, the full data migration device provided by the embodiments of the present application can be realized in a software manner, Figure 2The full-data migration device 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: an acquisition module 5551 and a migration module 5552, which are logical, and thus can be combined or further split according to the implemented functions.

[0061] The functions of the various modules will be described below.

[0062] In some embodiments of the present application, the device provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the device provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the full-data migration method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0063] The exemplary application and implementation of the server provided by the embodiments of the present application will be described in the embodiments of the present application, in which the server will be taken as the execution subject to describe the full-data migration method provided by the embodiments of the present application.

[0064] Referring to Figure 3 , Figure 3 is an optional flowchart of the full-data migration method provided by the embodiments of the present application, which will be described in combination with the steps shown in Figure 3 .

[0065] In step 301, full data in a source database is acquired; the full data includes at least one migration object.

[0066] In some embodiments of the present application, after the server receives the migration request sent by the administrator, the server determines the source database and the target database that need to be migrated according to the migration request. The migration request is used to request to migrate the full data in the source database to the target database. The full data is all the data in the source database.

[0067] In some embodiments of the present application, step 301 can be implemented by the server sending a database information acquisition request to the source database, the database information acquisition request being used to acquire full data information of the source database, the full data information including storage information of at least one migration object in the full data. The server can acquire the migration object from the source database according to the storage information of each migration object and take it out. The migration object can be a data table in the database.

[0068] In some embodiments of the present application, the server is in communication connection with at least one terminal. After receiving a database query request sent by any terminal, the server can query database information of at least one database connected to the server, send the database information of the at least one database to the terminal, and display the at least one database and the database information of each database through a manual interaction interface on the terminal side. The terminal can also receive a migration request of an administrator through the manual interaction interface, and determine the source database and the target database that need to be migrated according to the migration request.

[0069] In step 302, the migration object is extracted from at least one migration object of the full data as a to-be-migrated object in sequence, and the to-be-migrated object is migrated from the source database to the target database through an idle migration process in the at least one migration process.

[0070] In some embodiments of the present application, the server includes at least one migration process, and the at least one migration process can be executed in parallel. In some embodiments, one migration process can simultaneously execute migration operations of at least one to-be-migrated object. The migration process can provide a data migration service (DTS) and support services for data synchronization and transmission between multiple homogeneous or heterogeneous databases.

[0071] In some embodiments of the present application, one migration process can simultaneously execute migration operations of multiple to-be-migrated objects. The server can acquire a process load quantization value of each migration process. When the process load quantization value of the migration process is higher than or equal to a preset load threshold, it is determined that the process state of the migration process is a high load state. When the process load quantization value of the migration process is lower than the preset load threshold, it is determined that the migration process can still execute migration operations of to-be-migrated objects, and thus the process state of the migration process is determined to be a low load state. The migration process in the low load state is determined to be an idle migration process.

[0072] The load quantization value can be the number of to-be-migrated objects that are simultaneously migrated, or the resource utilization rate when the migration process processes each to-be-migrated object.

[0073] For example, there are three migration processes, namely a first process, a second process and a third process, and full data including first migration objects to fifth migration objects needs to be migrated. In the process of starting migration, the first process, the second process and the third process are all in a low load state, and thus are idle migration processes. At this time, according to the preset load threshold corresponding to each migration process, migration objects are extracted from the full data as to-be-migrated objects and are allocated to the first process to the third process, so that the first process to the third process change from the low load state to the high load state.

[0074] In the case that the load quantization value is the number of to-be-migrated objects to be migrated simultaneously, and the preset load threshold of each migration process is 1, the first migration object to the third migration object are extracted to obtain the first to-be-migrated object to the third to-be-migrated object, and the migration process of the first to-be-migrated object is performed by the first (idle) process, the migration process of the second to-be-migrated object is performed by the second (idle) process, and the migration process of the third to-be-migrated object is performed by the third (idle) process. At this time, each migration process is in a high load state.

[0075] In the case that the load quantization value is resource utilization and the preset load threshold of each migration process is 80%, the first migration object to the third migration object are extracted to obtain the first to-be-migrated object to the third to-be-migrated object, and the migration process of the first to-be-migrated object is performed by the first (idle) process, the migration process of the second to-be-migrated object is performed by the second (idle) process, and the migration process of the third to-be-migrated object is performed by the third (idle) process. At this time, the resource utilization of each migration process is obtained. If the resource utilization of the first process to the third process is 40%, 50% and 90% respectively, the first process and the second process are in a low load state and are still idle migration processes, and the third process is in a high load state. The fourth migration object and the fifth migration object need to be continuously extracted as the fourth to-be-migrated object and the fifth to-be-migrated object, and the migration process of the fourth to-be-migrated object is performed by the first process and the migration process of the fifth to-be-migrated object is performed by the second process.

[0076] In some embodiments of the present application, if each migration process in the at least one migration process is in a working state, the extraction of migration objects from the at least one migration object is stopped. Until there is at least one idle migration process.

[0077] In some embodiments of the present application, the server can obtain, according to the number of idle migration processes in the at least one migration process, the same number of migration objects as the number of processes from the at least one migration object in the source database, take the migration objects as the to-be-migrated objects, and migrate the same number of to-be-migrated objects from the source database to the target database through the at least one idle migration process. Each idle migration process performs a migration operation on one to-be-migrated object.

[0078] In some embodiments of the present application, the migration objects can be sequentially extracted from the at least one migration object of the full data as the to-be-migrated objects in the following ways: (1) copying the migration objects in the at least one migration object, taking the copied migration objects as the to-be-migrated objects, and adding a migrated label to the copied migration objects; in the subsequent migration process, migration objects are obtained from the migration objects without the migrated label for migration until all the migration objects in the full data are added with the migrated label; (2) directly taking the migration objects from the at least one migration object in the full data as the to-be-migrated objects; in the subsequent migration process, migration objects are obtained from the remaining migration objects in the full data for migration until there is no any migration object in the full data.

[0079] It should be noted that, compared with the scheme in the prior art that the full data is first fully imported into the read-write storage pool, and then the full data is fully exported to the target database after the full import process is completed, the migration process of each to-be-migrated object in the present application realizes full export and full import of the full data at the same time.

[0080] In step 303, when the at least one migration object is migrated to the target database, the full data migration process is completed.

[0081] In some embodiments of the present application, when the full data migration process is a copy migration, the full data migration process of the full data in the source database to the target database is completed when each migration object in the at least one migration object in the full data in the source database is added with the migrated label. When the full data migration process is a move migration, the full data migration process of the full data in the source database to the target database is completed when there is no any migration object in the full data in the source database.

[0082] Through the embodiments of the present application, the full data migration process is completed when the at least one migration object in the full data in the source database is migrated to the target database. Figure 3As can be seen from the above exemplary embodiments, the embodiments of the present application refine the full-volume data into multiple migration objects, and compared with the scheme in the prior art that the full-volume data is first read into a read-write storage pool and then written into a target database after the reading process is completed, since the full-volume data is refined into migration objects, each migration object can be directly migrated into the target database, and the read-write storage pool is omitted, so that the migration speed is improved, the storage cost is reduced, the operation cost is lowered, the component dependency is reduced, and the overall system reliability is improved. In addition, since the multiple migration processes are used to perform parallel migration on each migration object, the technical effect of simultaneously performing full-volume export and full-volume import on the full-volume data is achieved as a whole, and the overall migration efficiency of the full-volume data is improved.

[0083] Referring to Figure 4 , Figure 4 is an optional flowchart of the full-volume data migration method provided by the embodiments of the present application, based on Figure 3 , Figure 3 Step 302 in the foregoing method can be updated to steps 401 to 403, which will be described in combination with the steps shown in Figure 4 .

[0084] In step 401, in a case where there is an idle migration process in the at least one migration process, a migration object is extracted from the at least one migration object as a to-be-migrated object.

[0085] In some embodiments of the present application, the server can obtain the process state of each migration process, in a case where the migration process is performing the migration operation on the to-be-migrated object, the process state of the migration process is determined as a working state; in a case where the migration process has not started or has completed the migration operation on the to-be-migrated object, the process state of the migration process is determined as an idle state; and the migration process in the idle state is determined as an idle migration process.

[0086] In step 402, the to-be-migrated object is assigned to the idle migration process.

[0087] In some embodiments of the present application, the number of extracted migration objects (to-be-migrated objects) can be determined according to the number of idle migration processes. For one idle migration process, one migration object can be extracted as a to-be-migrated object and assigned to the idle migration process, so that the idle migration process completes the migration process on the to-be-migrated object.

[0088] For example, there are three migration processes, namely a first process, a second process and a third process, and full data including a first migration object to a fifth migration object needs to be migrated. In the process of starting migration, the first process, the second process and the third process are all in an idle state, and thus are idle migration processes. At this time, the first migration object to the third migration object can be extracted to obtain a first to-be-migrated object to a third to-be-migrated object, the migration process of the first to-be-migrated object is performed by the first (idle) process, the migration process of the second to-be-migrated object is performed by the second (idle) process, and the migration process of the third to-be-migrated object is performed by the third (idle) process. At this time, each migration process is in a working state, and thus the extraction of the migration object from the at least one migration object is temporarily stopped. If the second process has completed the migration process of the second to-be-migrated object, the second process is in an idle state at this time. At this time, the fourth migration object in the full data in the source data can be extracted as a fourth to-be-migrated object, and the migration process of the fourth to-be-migrated object is performed by the second (idle) process. Similarly, the migration process of the first migration object to the fifth migration object is completed.

[0089] In step 403, the to-be-migrated object is migrated from the source database to the target database by the idle migration process.

[0090] According to the above exemplary implementation of the embodiment of the application, Figure 4 According to the above exemplary implementation of the embodiment of the application,

[0091] In some embodiments of the application, referring to Figure 5 , Figure 5 is an optional flowchart of the full data migration method provided by the embodiment of the application, based on Figure 4 , Figure 4 The step 403 shown in the figure can be updated to steps 501 to 502.

[0092] In step 501, the to-be-migrated object is taken out from the source database by a reading thread in the idle migration process, and the to-be-migrated object is written into a data queue corresponding to the idle migration process in the memory.

[0093] In some embodiments of the present application, each migration process (idle migration process) corresponds to a data queue in the memory. In the migration process of the migration object by the migration process (idle migration process), the migration object is taken out from the source database and written into the data queue corresponding to the migration process (idle migration process) in the memory in an asynchronous manner. The data queue can store multiple migration objects at the same time, and the migration object is taken out from the data queue and written into the target database to complete the migration process of the migration object from the source database to the target database.

[0094] In some embodiments of the present application, each migration process (idle migration process) includes a read thread. The read thread is used to take out the migration object from the source database and write the migration object into the data queue corresponding to the idle migration process in the memory.

[0095] In some embodiments of the present application, the above step 501 can be implemented by the following steps: in step 5011, the structure data of the migration object is read from the source database to the data queue by the read thread; in step 5012, the source database data of the migration object is read from the source database to the data queue by the read process.

[0096] In some embodiments of the present application, each migration object in the full data includes source database data and structure data. The source database data is specific data in the full data of the source database that needs to be migrated to the target database. The structure data is the schema of the source database, which is the general term of schema objects. The schema objects include at least one of the following objects: table, column, data type, view, stored procedure, relationship, primary key, foreign key, etc.

[0097] In the above step 501, in the case that the data queue is first-in first-out, the structure data of the migration object needs to be obtained and written into the data queue first, so as to ensure that the target database is configured by the structure data first in the process of writing into the target database, and then the source database data of the migration object is written into the data queue, so that the target database has completed the configuration process according to the structure data of the migration object when the source database data is written into the target database. In the case that the data queue is first-in first-out, the source database data of the migration object can be obtained and written into the data queue first, and then the structure data is obtained and written into the data queue. In step 502, the write thread in the idle migration process takes out the migration object from the data queue and writes the migration object into the target database.

[0098] In some embodiments of the present application, each migration process (idle migration process) comprises a write thread, wherein the write thread is used to take the to-be-migrated object from the data queue and write the to-be-migrated object into the target database.

[0099] In some embodiments of the present application, in the process of taking the to-be-migrated object from the data queue and writing it into the target database, in order to improve the writing efficiency and writing accuracy, the structure data of the to-be-migrated object can be taken out first, and the structure data is written into the target database, so that the target database completes its configuration process according to the structure data. After configuration, the efficiency of absorbing source database data in the to-be-migrated object and the writing accuracy can be improved, and then the source database data of the to-be-migrated object is taken out and written into the target database configured.

[0100] In some embodiments of the present application, the above step 502 can be implemented by the following steps: in step 5021, the structure data of the to-be-migrated object is taken out from the data queue by the write thread, and the structure data is written into the target database to complete the configuration process of the target database according to the structure data; in step 5022, the source database data of the to-be-migrated object is taken out from the data queue by the write thread, and the source database data is written into the target database configured by the structure data.

[0101] In some embodiments of the present application, the read thread and the write thread in the migration process are executed in parallel. In the case where the migration process does not perform other migration tasks and only migrates one to-be-migrated object, the migration process can be divided into the following stages: in the first stage, the read thread obtains the structure data corresponding to the to-be-migrated process in the source database and writes it into the data queue, at this time, the write thread is in an idle state; in the second stage, the read thread has completed the process of writing the structure data into the data queue, at this time, the read thread starts to obtain the source database data of the to-be-migrated process in the source database and writes it into the data queue, and at the same time, the write thread writes the structure data in the data queue into the target database to configure the target database. It can be seen that in this stage, for the to-be-migrated object, the reading and writing processes can be executed in parallel; in the third stage, the write thread has completed the configuration process of the target database, and in the case where the read thread has written the source database data into the data queue, it starts to write the source database data into the target database which has been configured. At the same time, if there are still to-be-migrated objects in the source database which have not been migrated, the next to-be-migrated object is read by the read thread, at this time, the writing process of the previous to-be-migrated object and the reading process of the next to-be-migrated object are executed in parallel.

[0102] In some embodiments of the present application, the migration object in the full data can include source library data and structure data, can only include source library data, and can only include structure data. In the case where the migration object includes source library data and structure data, the read-write method in the above embodiments needs to be used to first migrate the structure data of the migration object to the target database, and then migrate the source library data of the migration object to the target database. In the case where the migration object only includes structure data, after the structure data of the migration object is migrated to the target database, the source library data of the migration object does not need to be migrated. In the case where the migration object only includes source library data, the structure data of the migration object does not need to be migrated, and the structure data of the migration object is directly migrated to the target database. On the basis of the present embodiment, the full data migration method provided by the present application can further include: obtaining a migration instruction of each migration object in the at least one migration object, and the migration instruction can be one of the following: a first migration instruction for simultaneously migrating source library data and structure data of the migration object; a second migration instruction for migrating source library data of the migration object and not migrating structure data of the migration object; and a third migration instruction for migrating structure data of the migration object and not migrating source library data of the migration object.

[0103] It can be known from the above exemplary implementation of the present application that, in the migration process of the to-be-migrated object from the source database to the target database, the data queue is used to buffer the to-be-migrated object, the asynchronous migration process of each to-be-migrated object is implemented, and the data migration efficiency is improved. In the migration process of each to-be-migrated object, the structure data of the to-be-migrated object is first migrated and the target database is configured, and then the source library data of the to-be-migrated object is migrated to the configured target database, so that the data migration efficiency and the migration accuracy can be further improved. Figure 5

[0104] In some embodiments of the present application, referring to Figure 6 , Figure 6 is an optional flowchart of the full data migration method provided by the present application, based on Figure 4 , the method can further include steps 601 to 602.

[0105] In step 601, the process state of each migration process in at least one migration process is monitored; the process state includes a read thread state and a write thread state.

[0106] ​In some embodiments of the present application, whether there is an idle migration process can be determined by detecting the process state of each migration process. According to the above embodiments, each migration process includes a read thread and a write thread, and the read thread and the write thread are executed in parallel, so the process state of each migration process can be determined by monitoring the read thread state of the read thread and the write thread state of the write thread in each migration process.

[0107] The process state of each migration process can include: the read thread state and the write thread state are both in a working state; the read thread state is in a working state and the write thread state is in an idle state; the write thread state is in a working state and the read thread state is in an idle state; and the read thread state and the write thread state are both in an idle state.

[0108] In step 602, an idle migration process is determined in at least one migration process according to the read thread state and the write thread state corresponding to each migration process.

[0109] In some embodiments of the present application, the above step 602 can include: step 6021, in the case that the read thread state and the write thread state corresponding to the migration process are both in an idle state, the migration process is determined as an idle migration process. Through this embodiment, it can be ensured that the migration process completes the migration task of a migration object, and then completes the migration task of the next migration object through the migration process, which can reduce the risk of errors during data migration and ensure the accuracy of data migration.

[0110] In some embodiments of the present application, the above step 602 can further include: step 6022, in the case that the read thread state corresponding to the migration process is in an idle state and the write thread state is in a working state, the migration process is determined as an idle migration process. Through this embodiment, in the case that the read rate of the source database and the write rate of the target database are uncontrollable, the risk of low data migration efficiency caused by fluctuations in the read rate and the write rate can be reduced, and the overall data migration efficiency of the full data is improved.

[0111] Through the embodiments of the present application, the process state of each migration process can be determined by monitoring the read thread state of the read thread and the write thread state of the write thread in each migration process, so that the idle migration process can be determined, and the risk of errors during data migration can be reduced, and the accuracy of data migration can be ensured. Figure 6As can be known from the foregoing exemplary embodiments, by detecting the thread state of the reading thread and the writing thread in each migration process, and then determining the idle migration thread according to the reading thread state and the writing thread state, the migration process can be ensured to complete the migration task of the next migration object after completing the migration task of one migration object, the risk of errors during data migration can be reduced, and the correctness of data migration can be ensured. In addition, in the case that the reading rate of the source database and the writing rate of the target database are uncontrollable, the risk of low data migration efficiency caused by fluctuations in the reading rate and the writing rate can be reduced, and the overall data migration efficiency of the full data is improved.

[0112] In some embodiments of the present application, referring to Figure 7 , Figure 7 is an optional flow diagram of the full data migration method provided by the embodiments of the present application, based on Figure 2 , the method can further include steps 701 to 703.

[0113] In step 701, the reading thread in the idle migration process takes the migration object to be migrated from the source database, and writes the migration object to be migrated into the data queue corresponding to the idle migration process in the memory; the writing thread in the idle migration process takes the migration object to be migrated from the data queue, and writes the migration object to be migrated into the target database.

[0114] In some embodiments of the present application, the specific description of the foregoing step 701 can refer to the description in the Figure 5 embodiments.

[0115] In step 702, in the process of taking the migration object to be migrated from the source database by the reading thread in the idle migration process, and writing the migration object to be migrated into the data queue corresponding to the idle migration process in the memory, if the data queue is full, the reading thread is suspended.

[0116] In some embodiments of the present application, for one migration process, in the case that the reading thread in the migration process is in an idle state, the migration process can be determined as an idle migration process, and the structure data and the source database data of the migration object to be migrated are continuously read from the source database and written into the data queue corresponding to the migration process. Since the data queue is located in the memory, its capacity is limited, and in the case that the writing rate of the writing thread is low, the reading thread may exceed the fixed capacity of the data queue in the process of reading the structure data and the source database data of the migration object to be migrated and writing them into the data queue, and therefore, the reading thread is suspended and only the writing thread is run in the case that the data queue is full.

[0117] In an embodiment of the present application, whether the data queue is full can be determined by the following method: obtaining the number of to-be-migrated objects temporarily stored in the data queue, and determining that the data queue is full when the number of to-be-migrated objects reaches a preset number threshold; obtaining the data amount of to-be-migrated objects temporarily stored in the data queue, and determining that the data queue is full when the ratio of the data amount to the capacity of the data queue exceeds a first ratio.

[0118] In step 703, in the process of taking to-be-migrated objects from the data queue by the write thread in the idle migration process and writing the to-be-migrated objects into the target database, the load quantization value in the data queue is obtained, and the read thread is resumed when the load quantization value is lower than a preset threshold.

[0119] In some embodiments of the present application, the read thread is temporarily suspended due to the full data queue, and the read thread can be resumed by obtaining the load quantization value in the data queue after the to-be-migrated objects are taken from the data queue by the write thread and written into the target database. The load quantization value can be the number of to-be-migrated objects, and the preset threshold is the preset number threshold. The read thread is resumed when the number of to-be-migrated objects is less than the preset number threshold. The load quantization value can also be the ratio of the data amount to the capacity of the data queue, and the preset threshold is a second ratio. The read thread is resumed when the ratio is lower than the second ratio, and the second ratio is less than or equal to the first ratio.

[0120] From the above exemplary implementation of the present application Figure 7 As can be seen from the above exemplary implementation of the present application, when the data queue corresponding to the migration process is full, the read thread of the migration process is suspended, which can ensure the stability of the data queue in the memory. At the same time, when the load quantization value of the data queue is lower than the preset threshold, the read thread is resumed in time, which can effectively improve the overall transmission efficiency of full data while ensuring stability.

[0121] In the following, an exemplary application of the present application in an actual application scenario will be described.

[0122] The embodiments of the present application propose a new migration process and architecture for the full export and full import stages of database data migration, which improves migration efficiency, saves migration time, improves stability, and reduces cost.

[0123] The database data migration full process is mainly divided into a full stage and an incremental stage. The full stage is divided into a full export stage and a full import stage. The incremental stage is divided into an incremental export stage and an incremental import stage. The inventor finds that structure data (schema) defines the organization form of data (data), and therefore, the structure data must be imported into the target database before the data is imported into the target database. The related scheme exports the database data to a storage pool, and then imports the database data from the storage pool into the target end after the export is completed.

[0124] The embodiment of the application optimizes the above process as follows:

[0125] (1) The full import does not need to wait for all migration objects to start after the full export is completed. The migration task full stage scheduling granularity is refined to a specific migration object in the scheme. When the migration object imports the structure data from the source end into the target end, the data read from the source end can be imported into the target end at the same time. In this way, the full export and the full import are executed in parallel from a macro perspective, and the overall migration efficiency is improved.

[0126] (2) In the full export stage, the structure data and the data do not need to enter the storage pool, but are directly imported into the target database. The reading and writing of the storage pool are removed, which not only improves the migration speed, but also reduces the storage overhead and the operating cost. The reduction of component dependency also improves the overall reliability of the system. The storage pool in the related technology refers to a disk, a disk array, an object storage system, or a message queue.

[0127] In the related technology, the data migration of the database, for example, logical export based on mysqldump, mydumper, or physical export based on xtrabackup, separates the full export and the full import stages. That is, the full import cannot be started until the full export of all migration objects is completed, and there is a global synchronization barrier. The process is as shown in Figure 8 .

[0128] Referring to the migration process diagram in Figure 8 , the amount of data exported in the full export stage is huge. Due to the existence of this synchronization barrier, the data cannot be cached in the memory. Therefore, in the full export stage, the exported information must be placed in the external storage for the full import stage to read. The exported data can exist in the local disk, the object storage, the cloud hard disk, etc. The application abstracts it as a storage pool.

[0129] The applicant finds that the related technology has the following disadvantages:

[0130] (1) Full export and full import cannot be performed simultaneously. When the source database is exported, the target database is completely idle and waits for the export to complete. For example, there is a database with a capacity of 6T, and in an absolutely ideal case (no user load, all resources are full load for data migration), it takes about 18 hours to export 6T data. Full import is a process of execution and writing, which will take much longer than 18 hours. Therefore, there is great optimization value for this process.

[0131] (2) Export needs to be written to the storage pool, and import reads from the storage pool. The performance of the storage pool limits the performance of full import and export. The space of the storage pool must be larger than the source database instance, increasing the economic cost.

[0132] (3) Full export and import take too long, increasing the entire data migration time and cost. Since incremental data export starts from the initiation of migration, incremental data also needs to be stored in the storage pool. Since the remote database is readable and writable during the data migration process, the source database changes frequently, which is a very common scenario, and the incremental data generated between full export and full import will be large. Incremental data will occupy storage resources for a long time, and will also increase the workload and time of the incremental import stage.

[0133] To address this deficiency, the applicant removes this synchronization barrier: full export and import are merged, reducing the overall time from the process. After removing the synchronization barrier, the storage pool is no longer optional, and data is directly "flowed" from the source to the target database. This improves performance, reduces economic cost, and improves system reliability.

[0134] Please refer to Figure 9 , which shows an optional interactive interface diagram of a database migration service. In the interactive interface 90, multiple migration tasks are displayed, and the administrator can perform operations such as viewing, modifying, checking, starting, deleting, canceling, and data comparison on each migration task, for example, after selecting one of the migration tasks 91, a secondary menu diagram as shown in Figure 10 can be obtained, in part of the interface 911, the secondary menu 912 corresponding to the migration task 91 can be obtained, and the administrator's selection operation on the "full migration" option of the secondary menu 912 can be detected to obtain the administrator's request for migrating full data.

[0135] In some embodiments of the present application, please refer to Figure 11 , Figure 11 is an optional migration process diagram of the full data migration method provided by the embodiments of the present application.

[0136] This scheme merges full export and full import into two modules: a management module 111 and an execution module 112. Among them:

[0137] The management module 111: knows the objects to be migrated, for example, which specific library table. The objects to be migrated are encapsulated into jobs to form a job queue, which is dispatched to the execution module 112 asynchronously. When there is an idle execution module, the job is dispatched to the execution module; when the job queue is empty, that is, the job is completed, the full export and import are completed. Each table is an object to be migrated. One object to be migrated is divided into a "structure data part" (also called schema, that is, structure data) and a "data part" (also called data, that is, the specific data in the source library). The migration instruction of the administrator for each object to be migrated can be received, which can include a first migration instruction for simultaneously migrating the "structure data part" and the "data part", a second migration instruction for only migrating the "data part", and a third migration instruction for only migrating the "structure data part". If no migration instruction of the administrator for each object to be migrated is received, the "structure data part" and the "data part" are migrated by default. If the administrator wants to migrate only the "structure data part" of the object to be migrated to the target database without specific data, the third migration instruction can be selected; if the data structure of the target database has been established, and the administrator wants to migrate only the "data part" of the object to be migrated to the target database, the second migration instruction can be selected.

[0138] The execution module 112: the execution module has multiple entities (programming term instance. It can be understood as a process or a service. Any number of execution modules can be owned, the more the better for migration; but the more the higher the cost), which is initialized to complete the connection of the source DB and the target DB, and then waits for the job dispatch of the management module 111. When the job is received, the structure data of the object to be migrated in the job is read from the source DB and sent to the target DB for execution, and the structure data is constructed in the target DB. Then, the source library data of the object to be migrated is read from the source DB and sent to the target DB for execution, and the source library data is constructed in the target DB. When the source library data construction is completed, the execution module completes the full export and full import of the object to be migrated, and can execute a new job to perform full export and import on other migration objects.

[0139] The above steps scatter the global synchronization barrier object in the traditional scheme: all objects to be migrated, into various jobs, which are executed in parallel by multiple execution modules without waiting for all objects to complete a certain stage. This is a parallelization transformation from the global task allocation stage, which improves the performance.

[0140] In some embodiments of the present application, see Figure 12 , Figure 12 is an optional migration process schematic diagram of the full data migration method provided by the embodiments of the present application.

[0141] The dump thread 121 of the execution module continuously reads the structure data or data from the source DB and writes into the high-speed memory data queue asynchronously, which replaces the storage pool in the traditional scheme. The asynchronous mode ensures that the structure data or data is continuously obtained from the source end when the queue is not full. When the queue is full, the acquisition action is suspended until the queue has space to allow writing. The dump thread and the job are in one-to-one correspondence. One thread processes one job, which can achieve the best performance.

[0142] At the same time, the apply thread 122 of the execution module reads the structure data or data from the queue and writes to the target DB until the dump thread completes the work and the queue is empty.

[0143] Among them, the source DB will have user load, and the dump speed may be fast or slow, and the existence of the high-speed memory queue smooths the speed change. It is beneficial to improve the overall efficiency. The implementation mode of the execution module removes the synchronous barrier of exporting full data and importing full data from the data migration process of each migration object, and also removes the dependence on the storage pool. The use of the storage pool is removed, the performance is improved, the cost is reduced, the component dependence is reduced, and the system reliability is improved.

[0144] The full data migration method provided by the above embodiment can achieve the following technical effects: (1) improve the migration speed; after removing the synchronous barrier of full data export and full data import, each migration object can be independent, without waiting, to complete full data migration. Removing the storage pool reduces the negative impact of storage pool performance on migration speed. The full data migration in the full data stage can fully utilize the network speed and DB performance, and reduce the speed constraints of unnecessary components. (2) improve the reliability and reduce the component dependence; removing the storage pool reduces the complexity of the components and improves the reliability. (3) reduce the operating cost; removing the storage pool does not need to provide storage space for full data transfer for each migration task. Moreover, the migration speed in the full data stage is improved, which also reduces the storage space required for incremental data. The above factors are superimposed to greatly reduce the operating cost.

[0145] The following continues to illustrate an exemplary structure of the full data migration device 555 implemented as a software module provided by the embodiments of the present application. As shown in Figure 2 the software modules stored in the full data migration device 555 of the storage 550 can include:

[0146] The acquisition module 5551 is configured to acquire full data in the source database; the full data includes at least one migration object;

[0147] The migration module 5552 is configured to sequentially extract a migration object from at least one migration object of the full data as a to-be-migrated object, and migrate the to-be-migrated object from the source database to the target database through an idle migration process in the at least one migration process.

[0148] The migration module 5552 is further configured to complete the full data migration process when all the at least one migration object is migrated to the target database.

[0149] In some embodiments of the present application, the migration module 5552 is further configured to extract a migration object from the at least one migration object as a to-be-migrated object when there is an idle migration process in the at least one migration process; assign the to-be-migrated object to the idle migration process; and migrate the to-be-migrated object from the source database to the target database through the idle migration process.

[0150] In some embodiments of the present application, the migration module 5552 is further configured to take out the to-be-migrated object from the source database through a reading thread in the idle migration process, and write the to-be-migrated object into a data queue corresponding to the idle migration process in the memory; and take out the to-be-migrated object from the data queue through a writing thread in the idle migration process, and write the to-be-migrated object into the target database.

[0151] In some embodiments of the present application, the migration module 5552 is further configured to read structure data of the to-be-migrated object from the source database to the data queue through the reading thread; and read source database data of the to-be-migrated object from the source database to the data queue through the reading process.

[0152] In some embodiments of the present application, the migration module 5552 is further configured to take out the structure data of the to-be-migrated object from the data queue through the writing thread, and write the structure data into the target database to complete a configuration process of the target database according to the structure data; and take out the source database data of the to-be-migrated object from the data queue through the writing thread, and write the source database data into the target database configured by the structure data.

[0153] In some embodiments of the present application, the migration module 5552 is further configured to monitor a process state of each migration process in the at least one migration process; the process state includes a reading thread state and a writing thread state; and determine the idle migration process in the at least one migration process according to the reading thread state and the writing thread state corresponding to each migration process.

[0154] In some embodiments of the present application, the determining the idle migration process according to the read thread state and the write thread state corresponding to each migration process in at least one migration process comprises at least one of the following: determining the migration process as the idle migration process in the case that the read thread state and the write thread state corresponding to the migration process are all in the idle state; determining the migration process as the idle migration process in the case that the read thread state corresponding to the migration process is in the idle state and the write thread state corresponding to the migration process are all in the working state.

[0155] In some embodiments of the present application, the migration module 5552 is further configured to, in the process of taking the to-be-migrated object from the source database by the read thread in the idle migration process and writing the to-be-migrated object into the data queue corresponding to the idle migration process in the memory, suspend the read thread in the case that the data queue is full; in the process of taking the to-be-migrated object from the data queue by the write thread in the idle migration process and writing the to-be-migrated object into the target database, acquire the load quantization value in the data queue, and resume the read thread in the case that the load quantization value is lower than a preset threshold.

[0156] The embodiment of the present disclosure provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the full-amount data migration method provided in the embodiment of the present application.

[0157] The embodiment of the present disclosure provides a computer readable storage medium storing executable instructions, wherein the executable instructions, when executed by a processor, will cause the processor to perform the full-amount data migration method provided in the embodiment of the present application, for example, as shown in the method. Figures 3 to 7

[0158] In some embodiments of the present application, the computer readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or various devices comprising one or any combination of the above memories.

[0159] In some embodiments of the present application, the executable instructions can be in the form of a program, software, software module, script or code, written in any form of programming language (including a compiled or interpreted language, or a declarative or procedural language), and can be deployed in any form, including being deployed as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0160] ​As an example, executable instructions can correspond to a file in a file system, but are not limited thereto. The executable instructions can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language e.g., Hyper Text Markup Language (HTML), document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or code portions).

[0161] As an example, executable instructions can be deployed to execute on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected through a communication network.

[0162] To sum up, through the embodiments of the present application, the following technical effects can be achieved:

[0163] (1) The embodiments of the present application refine the full volume data to multiple migration objects. Compared with the scheme in the prior art that reads the full volume data to the read-write storage pool first and then writes the full volume data to the target database after the reading process is completed, since the refinement to the migration objects, each migration object can be directly migrated to the target database, and the read-write storage pool is saved. Not only the migration speed is improved, but also the storage cost is reduced, the operation cost is lowered, the component dependency is reduced, and the overall system reliability is improved. Moreover, since the multiple migration processes are used to perform parallel migration on each migration object, the technical effect of simultaneously performing full volume export and full volume import on the full volume data is achieved as a whole, and the overall migration efficiency of the full volume data is improved.

[0164] (2) The embodiments of the present application complete the migration process of the to-be-migrated object by the idle migration process in the at least one migration process, which can ensure the migration efficiency of the to-be-migrated object, avoid the problem that the overall data migration efficiency of the full volume data is low due to uneven allocation of multiple migration objects, and improve the overall migration efficiency of the full volume data.

[0165] (3) In the migration process of the to-be-migrated object from the source database to the target database, the embodiments of the present application use the data queue to buffer the to-be-migrated object, which realizes the asynchronous migration process of each to-be-migrated object and improves the data migration efficiency. Moreover, in the migration process of each to-be-migrated object, the structure data of the to-be-migrated object is migrated first, the target database is configured, and then the source database data of the to-be-migrated object is migrated to the configured target database, which can further improve the data migration efficiency and the migration accuracy.

[0166] (4) By detecting the thread state of the reading thread and the writing thread in each migration process, and then determining the idle migration thread according to the reading thread state and the writing thread state, it can be ensured that the migration process completes the migration task of a migration object, and then completes the migration task of the next migration object through the migration process, which can reduce the risk of errors during data migration, ensure the correctness of data migration, and in the case that the reading rate of the source database and the writing rate of the target database are uncontrollable, reduce the risk of low data migration efficiency caused by fluctuations in the reading rate and the writing rate, and improve the overall data migration efficiency of the full data.

[0167] (5) In the case that the data queue corresponding to the migration process is full, the reading thread of the migration process is suspended, which can ensure the stability of the data queue in the memory; at the same time, in the case that the load quantization value of the data queue is lower than the preset threshold, the reading thread is restored in time, which can effectively improve the overall transmission efficiency of the full data while ensuring the stability.

[0168] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A full data migration method, characterized by, The method comprises: acquiring full data in a source database; the full data comprises at least one migration object; in the case that there is an idle migration process in at least one migration process, extracting a migration object from the at least one migration object as a to-be-migrated object; allocating the to-be-migrated object to the idle migration process; taking out the to-be-migrated object from the source database by a reading thread in the idle migration process, and writing the to-be-migrated object into a data queue corresponding to the idle migration process in the memory; taking out the to-be-migrated object from the data queue by a writing thread in the idle migration process, and writing the to-be-migrated object into a target database, wherein when the source database data of the to-be-migrated object is written into the target database, the target database has completed a configuration process according to the structure data of the to-be-migrated object; in the case that the at least one migration object is all migrated to the target database, completing the full data migration process.

2. The method of claim 1, wherein, The taking out of the to-be-migrated object from the source database by the reading thread in the idle migration process, and the writing of the to-be-migrated object into the data queue corresponding to the idle migration process in the memory, comprises: reading the structure data of the to-be-migrated object from the source database to the data queue by the reading thread; reading the source database data of the to-be-migrated object from the source database to the data queue by the reading process.

3. The method according to claim 1 or 2, characterized in that, The taking out of the to-be-migrated object from the data queue by the writing thread in the idle migration process, and the writing of the to-be-migrated object into the target database, comprises: taking out the structure data of the to-be-migrated object from the data queue by the writing thread, and writing the structure data into the target database, so as to complete the configuration process of the target database according to the structure data; taking out the source database data of the to-be-migrated object from the data queue by the writing thread, and writing the source database data into the target database configured by the structure data.

4. The method of claim 1, wherein, The method further comprises: monitoring the process state of each migration process in the at least one migration process; the process state comprises a reading thread state and a writing thread state; determining the idle migration process in the at least one migration process according to the reading thread state and the writing thread state corresponding to each migration process.

5. The method of claim 4, wherein, The determination of the idle migration process in the at least one migration process according to the reading thread state and the writing thread state corresponding to each migration process comprises at least one of the following: in the case that the reading thread state and the writing thread state corresponding to the migration process are all idle states, determining the migration process as the idle migration process; in the case that the reading thread state corresponding to the migration process is an idle state, and the writing thread state is all working states, determining the migration process as the idle migration process.

6. A total data migration apparatus, characterized by comprising: The method comprises: an acquisition module, configured to acquire full data in a source database; the full data comprises at least one migration object; a migration module configured to extract a migration object as a to-be-migrated object from the at least one migration object when there is an idle migration process in the at least one migration process; and assign the to-be-migrated object to the idle migration process; read the to-be-migrated object from the source database by a reading thread in the idle migration process, and write the to-be-migrated object into a data queue corresponding to the idle migration process in the memory; and write the to-be-migrated object from the data queue by a writing thread in the idle migration process, and write the to-be-migrated object into a target database, wherein the target database has completed a configuration process according to structure data of the to-be-migrated object when writing source database data of the to-be-migrated object into the target database. The migration module is further configured to complete the full-data migration process when the at least one migration object is migrated to the target database.

7. The apparatus of claim 6, wherein The migration module is further configured to read the structure data of the to-be-migrated object from the source database to the data queue by the reading thread; and read the source database data of the to-be-migrated object from the source database to the data queue by the reading process.

8. The apparatus of claim 6 or 7, wherein The migration module is further configured to take the structure data of the to-be-migrated object from the data queue by the writing thread, and write the structure data into the target database to complete a configuration process of the target database according to the structure data; and take the source database data of the to-be-migrated object from the data queue by the writing thread, and write the source database data into the target database configured by the structure data.

9. The apparatus of claim 6, wherein The migration module is further configured to monitor a process state of each of the migration processes in the at least one migration process; the process state includes a reading thread state and a writing thread state; and determine the idle migration process in the at least one migration process according to the reading thread state and the writing thread state corresponding to each of the migration processes.

10. A full volume data migration device, characterized by, including: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the method of any one of claims 1 to 5.

11. A computer readable storage medium, characterized in that, executable instructions stored in the memory, configured to be executed by a processor to implement the method of any one of claims 1 to 5.

12. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by a processor to implement the method of any one of claims 1 to 5.

Images