A data migration method and device, computer equipment and storage medium

Abstract

Embodiments of the present application disclose a data migration method and device, computer equipment and a storage medium. The present application obtains data migration requirements of a business for heterogeneous database migration. Based on the data migration requirements, a data migration strategy is configured for a preset database middleware. A data extraction process is created on the database middleware, and target business data is obtained from a source database through the data extraction process. A data replication process is created on the database middleware, and the target business data is data-sharded according to the data migration strategy through the data replication process, and each target business sub-data after data-sharding is migrated to a corresponding position in a target database, thereby improving the efficiency of data migration.

Description

Technical Field

[0001] This application relates to the field of data migration technology, specifically to a data migration method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of society, more and more companies are adopting heterogeneous database migration to move data from one source database to another in order to facilitate data management and better carry out related financial business.

[0003] The existing process of migrating data from one database to another can only be based on standard protocols. However, since the data volume and type of different financial businesses in an enterprise are different, using specific standard protocols for data migration often cannot meet business needs well, resulting in low efficiency during data migration. Summary of the Invention

[0004] This application provides a data migration method, apparatus, computer equipment, and storage medium that can be used in financial technology or other related fields, thereby improving the efficiency of data migration.

[0005] This application provides a data migration method, including:

[0006] Obtain the business's data migration requirements when migrating heterogeneous databases;

[0007] Based on the above data migration requirements, configure a data migration strategy for the preset database middleware;

[0008] Create a data extraction process on the aforementioned database middleware, and obtain the target business data from the source database through the aforementioned data extraction process;

[0009] A data replication process is created on the aforementioned database middleware. Through this process, the target business data is sharded according to the aforementioned data migration strategy, and the sharded target business sub-data is migrated to the corresponding location in the target database.

[0010] Accordingly, embodiments of this application also provide a data migration apparatus, including:

[0011] The requirement elicitation module is used to obtain the data migration requirements of the business when migrating to heterogeneous databases.

[0012] The strategy configuration module is used to configure data migration strategies for the preset database middleware based on the above data migration requirements.

[0013] The data acquisition module is used to create a data extraction process on the aforementioned database middleware, and to obtain target business data from the source database through the aforementioned data extraction process.

[0014] The data migration module is used to create a data replication process on the aforementioned database middleware, and through the aforementioned data replication process, to shard the aforementioned target business data according to the aforementioned data migration strategy, and to migrate the sharded target business sub-data to the corresponding location in the target database.

[0015] Accordingly, embodiments of this application also provide a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes any of the data migration methods provided in embodiments of this application.

[0016] Accordingly, embodiments of this application also provide a storage medium storing multiple instructions adapted for loading by a processor to execute the data migration method described above.

[0017] This application embodiment obtains the data migration requirements of a business when migrating to a heterogeneous database; based on the data migration requirements, it configures a data migration strategy for a preset database middleware; it creates a data extraction process on the database middleware and obtains target business data from the source database through the data extraction process; it creates a data replication process on the database middleware and shards the target business data according to the data migration strategy through the data replication process, and migrates each sharded target business sub-data to the corresponding position in the target database. Thus, in the process of migrating data from the source database to another database, a database middleware is set up, and a data migration strategy is configured for the database middleware based on the data migration requirements of the business when migrating to a heterogeneous database, so as to effectively meet the business requirements when migrating data from the source database to another database, thereby improving the efficiency of data migration. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a data migration method provided in an embodiment of this application.

[0020] Figure 2This is a structural block diagram of a data migration device provided in an embodiment of this application.

[0021] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] This application provides a data migration method, apparatus, storage medium, and computer device. Specifically, the data migration method of this application can be executed by a computer device, which can be a server or a terminal, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, desktop computer, laptop, tablet computer, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0024] For example, the computer device can be a terminal that can acquire data migration requirements for heterogeneous database migration; based on the data migration requirements, configure a data migration strategy for a preset database middleware; create a data extraction process on the database middleware, and obtain target business data from the source database through the data extraction process; create a data replication process on the database middleware, and shard the target business data according to the data migration strategy through the data replication process, and migrate each sharded target business sub-data to the corresponding position in the target database.

[0025] To address the aforementioned issues, this application provides a data migration method, apparatus, computer device, and storage medium, which can improve the efficiency of data migration.

[0026] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0027] This application provides a data migration method, which can be executed by a terminal or a server. This application uses the execution of the data migration method by a terminal as an example for illustration.

[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating a data migration method provided in an embodiment of this application. The specific flow of the data migration method is as follows:

[0029] 101. Obtain the business's data migration requirements when migrating heterogeneous databases.

[0030] The aforementioned data migration requirements are the needs of different financial businesses when migrating to heterogeneous databases. By configuring corresponding data migration requirements for different financial businesses, the needs of different financial businesses have been met. For example, some financial businesses require that the data to be migrated be broken up during the data migration process so that the data can be migrated horizontally from the source database to another database.

[0031] For example, since some financial transactions involve order numbers, the aforementioned data migration requirement can be a requirement based on order numbers. Furthermore, the data migration requirement can be further designed with spatial information such as the quantity and location of the storage space after the data is migrated to another database. For example, after the data is migrated to another database, the data can be divided into two groups of sub-data, and the two groups of sub-data can be stored in two storage spaces in the other database respectively.

[0032] Specifically, the business application corresponding to the business can connect to the source database of the data to be migrated and the target database to which the data is migrated through a data access proxy. In other words, a proxy layer is built between the business application and the database for business processing, thereby enabling the interaction between the business application and the database.

[0033] 102. Based on the above data migration requirements, configure a data migration strategy for the preset database middleware.

[0034] In this embodiment, since different services have different data migration requirements, after obtaining the data migration requirements corresponding to the services, the terminal can provide these requirements and configure a data migration strategy for the preset database middleware. This allows the database middleware to use the data migration strategy to migrate data from the source database to another database, thereby making the data migration more in line with the service requirements. The data migration strategy includes a data sharding mechanism and a number of data shards. The data sharding mechanism is a mechanism for sharding data, and the number of data shards is the number of shards required when sharding the data.

[0035] For example, if the source database is an Oracle database and the target database after data migration is a MySQL database, a data migration strategy can be configured for this data migration. This involves migrating data from Oracle to MySQL using a 1->N or M->N (where N is greater than M and greater than 1) sharding approach. For instance, if the data in Oracle exists as a table (denoted as `t_order`), the `t_order` table can be heterogeneously migrated and horizontally sharded across four different `t_order` tables in the MySQL instance. That is, if there are four MySQL instances, MySQL1, MySQL2, MySQL3, and MySQL4, the `t_order` table can be heterogeneously migrated and horizontally sharded across these instances, resulting in a total of four shards. This horizontal sharding can be based on the aforementioned sharding mechanism, such as horizontally sharding the data in the `t_order` table using the primary key (denoted as `order_id`) as a modulo operation.

[0036] The structure of the t_order table mentioned above is as follows:

[0037] CREATE TABLE`t_order`(`order_id`int NOT NULL,`user_id`int DEFAULTNULL,`description`varchar(200)DEFAULT NULL,PRIMARY KEY(`order_id`))

[0038] The user_id and description mentioned above are fields.

[0039] Specifically, the aforementioned database middleware can be ShardingSphere middleware, and data migration strategies can be configured for this middleware, for example, by configuring a YAML file.

[0040] 103. Create a data extraction process on the aforementioned database middleware, and obtain the target business data from the source database through the aforementioned data extraction process.

[0041] In this embodiment, the terminal creates a data extraction process on the aforementioned database middleware to obtain target business data from the source database and stores the target business data in the message storage middleware corresponding to the database middleware.

[0042] In some embodiments, the target business data includes incremental business data and full business data, the data extraction process includes an incremental extraction process and a full extraction process, and obtaining the target business data from the source database may include: the terminal creating the incremental extraction process on the database middleware to obtain the business logs corresponding to the business from the source database through the incremental extraction process, and parsing the business logs to obtain the incremental business data; and obtaining the full business data corresponding to the business from the source database through the full extraction process.

[0043] The aforementioned incremental business data includes, but is not limited to, insert data, update data, or delete data.

[0044] In some embodiments, the message storage middleware corresponding to the database middleware includes an incremental data message set and a full data message set, and further includes: configuring an incremental identifier for the incremental data message set and configuring a full identifier for the full data message set; storing the incremental business data in the incremental data message set based on the incremental identifier; and storing the full business data in the full data message set based on the full identifier.

[0045] Specifically, when creating the above-mentioned incremental extraction process, an incremental data message set needs to be created synchronously on the message storage middleware, and an incremental identifier needs to be configured for the incremental data message set so as to store the acquired incremental business data into the incremental data message set in a timely manner.

[0046] The incremental identifier and the full identifier are used to identify the incremental data message set and the full data message set, respectively. For example, the incremental identifier can be set to incr_instance_name_database_table_name; the full identifier can be set to full_instance_name_database_table_name.

[0047] Specifically, a corresponding message storage middleware (denoted as Kafka) can be created based on the database middleware. This message storage middleware contains multiple message sets (denoted as topics), and the aforementioned message sets containing incremental data and full data are located in this message storage middleware.

[0048] 104. Create a data replication process on the aforementioned database middleware, and through the aforementioned data replication process, shard the aforementioned target business data according to the aforementioned data migration strategy, and migrate the sharded target business sub-data to the corresponding positions in the target database.

[0049] In this embodiment, the terminal consumes the target business data obtained in the data extraction process, i.e., the data in the message set, through the data replication process and the data migration strategy. Specifically, the target business data is sharded according to the data migration strategy, and each sharded target business sub-data is migrated to the corresponding position in the target database. Thus, during the data migration process, not only is the database burden reduced through data sharding, but also the inability to meet business needs due to insufficient database performance is avoided, thereby reducing business risks and achieving data migration with high efficiency.

[0050] In some embodiments, the data migration strategy includes a data sharding mechanism and a number of data shards. Sharding the target business data according to the data migration strategy may include: sharding the target business data according to the data sharding mechanism to obtain the target business sub-data of the specified number of data shards. Migrating each sharded target business sub-data to its corresponding location in the target database may include: migrating the target business sub-data of the specified number of data shards to their respective storage spaces in the target database based on the data migration strategy.

[0051] For example, the data sharding mechanism in the above data migration strategy can be set to horizontally shard the data by hashing the data according to the primary key.

[0052] In some embodiments, the data replication process includes an incremental replication process and a full replication process. The process of sharding the target business data according to the data migration strategy and migrating each sharded target business sub-data to its corresponding location in the target database via the data replication process includes: obtaining the full business data from the full data message set via the full replication process, sharding the full business data according to the data migration strategy, and migrating each sharded full business sub-data to its corresponding location in the target database; and obtaining the incremental business data from the incremental data message set via the incremental replication process, sharding the incremental business data according to the data migration strategy, and migrating each sharded incremental business sub-data to its corresponding location in the target database.

[0053] In some embodiments, the above-mentioned migration of each sharded target business sub-data to the corresponding position in the target database includes: creating multiple concurrent threads based on the data volume of the target business data through the above-mentioned data replication process; and allocating each sharded target business sub-data to each concurrent thread so that each concurrent thread migrates the allocated target business sub-data to the corresponding position in the target database.

[0054] The terminal can set a data sending threshold for each thread. When the amount of data in a thread meets the threshold, the target business sub-data allocated to that thread is migrated to the corresponding location in the target database. This data sending threshold can be set according to requirements, for example, to 1000 data entries.

[0055] In some embodiments, the method further includes: during the migration of the sharded target business sub-data to the target database, based on the data mapping relationship corresponding to the field types in the target business sub-data, performing corresponding data processing on the data in each field of the target business sub-data to ensure that the data between fields can be migrated to the target database normally. The data processing may include data format conversion, such as converting insert data into merge-in data.

[0056] This application discloses a data migration method, which includes: obtaining data migration requirements for heterogeneous database migration by a business; configuring a data migration strategy for a preset database middleware based on the data migration requirements; creating a data extraction process on the database middleware and obtaining target business data from the source database through the data extraction process; creating a data replication process on the database middleware and sharding the target business data according to the data migration strategy through the data replication process, and migrating each sharded target business sub-data to the corresponding position in the target database. Thus, during the migration of data from the source database to another database, a database middleware is set up, and a data migration strategy is configured for the database middleware based on the data migration requirements for heterogeneous database migration by the business, so as to effectively meet business needs when migrating data from the source database to another database, thereby improving the efficiency of data migration.

[0057] To facilitate better implementation of the data migration method provided in this application, this application also provides a data migration apparatus based on the above-described data migration method. The meanings of the terms used are the same as in the data migration method described above, and specific implementation details can be found in the descriptions within the method embodiments.

[0058] Please see Figure 2 , Figure 2 A structural block diagram of a data migration apparatus provided in this application embodiment, the apparatus comprising:

[0059] The requirement acquisition module 201 is used to acquire the data migration requirements of the business when migrating heterogeneous databases.

[0060] The strategy configuration module 202 is used to configure a data migration strategy for the preset database middleware based on the above data migration requirements.

[0061] The data acquisition module 203 is used to create a data extraction process on the aforementioned database middleware, and to obtain target business data from the source database through the aforementioned data extraction process.

[0062] The data migration module 204 is used to create a data replication process on the aforementioned database middleware, and through the aforementioned data replication process, to shard the aforementioned target business data according to the aforementioned data migration strategy, and to migrate the sharded target business sub-data to the corresponding positions in the target database.

[0063] In some embodiments, the above data migration strategy includes a data sharding mechanism and a number of data shards, and the data migration module 204 is specifically used for:

[0064] The target business data is sharded according to the data sharding mechanism described above to obtain the target business sub-data of the above number of data shards.

[0065] Based on the above data migration strategy, the target number of business sub-business data in the above data shards will be migrated to the corresponding storage space in the target database.

[0066] In some embodiments, the target business data includes incremental business data and full business data, the data extraction process includes an incremental extraction process and a full extraction process, and the data acquisition module 203 is specifically used for:

[0067] The incremental extraction process described above retrieves the business logs corresponding to the business from the source database and parses and processes these business logs to obtain the incremental business data.

[0068] The full data of the business is obtained from the source database through the full extraction process described above.

[0069] In some embodiments, the message storage middleware corresponding to the database middleware includes an incremental data message set and a full data message set, and the data migration device further includes a storage module, which is specifically used for:

[0070] Configure an incremental identifier for the above-mentioned incremental data message set, and configure a full identifier for the above-mentioned full data message set;

[0071] Based on the aforementioned incremental identifier, the aforementioned incremental business data is stored in the aforementioned incremental data message set;

[0072] Based on the aforementioned full identifier, the aforementioned full business data is stored in the aforementioned full data message set.

[0073] In some embodiments, the data replication process includes an incremental replication process and a full replication process, and the data migration module 204 is specifically used for:

[0074] The full business data is obtained from the full data message set through the full replication process described above. The full business data is then sharded according to the data migration strategy described above, and each sharded full business sub-data is migrated to the corresponding position in the target database described above.

[0075] The incremental business data is obtained from the incremental data message set through the incremental replication process, the incremental business data is sharded according to the data migration strategy, and the sharded incremental business sub-data is migrated to the corresponding position in the target database.

[0076] In some embodiments, the data migration module 204 is specifically used for:

[0077] Through the aforementioned data replication process, multiple concurrent threads are created based on the data volume of the target business data.

[0078] The sharded target business sub-data is allocated to each concurrent thread so that each concurrent thread can migrate the allocated target business sub-data to the corresponding location in the target database.

[0079] In some embodiments, the data migration apparatus further includes a data processing module, which is specifically used for:

[0080] During the migration of the sharded target business sub-data to the target database, the data in each field of the target business sub-data is processed accordingly based on the data mapping relationship corresponding to the field types in the target business sub-data.

[0081] This application discloses a data migration apparatus. A requirement acquisition module 201 acquires data migration requirements for heterogeneous database migration. A strategy configuration module 202 configures a data migration strategy for a preset database middleware based on these requirements. A data acquisition module 203 creates a data extraction process on the database middleware and acquires target business data from the source database through this process. A data migration module 204 creates a data replication process on the database middleware, shards the target business data according to the data migration strategy, and migrates each sharded sub-database to its corresponding location in the target database. This improves the efficiency of data migration.

[0082] Accordingly, embodiments of this application also provide a computer device, which can be a terminal. For example... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device 300 includes a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, and a computer program stored in the memory 302 and executable on the processor. The processor 301 and the memory 302 are electrically connected. Those skilled in the art will understand that the computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0083] The processor 301 is the control center of the computer device 300. It connects various parts of the computer device 300 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it performs various functions of the computer device 300 and processes data, thereby monitoring the computer device 300 as a whole.

[0084] In this embodiment, the processor 301 in the computer device 300 loads the instructions corresponding to the processes of one or more applications into the memory 302 according to the following steps, and the processor 301 runs the applications stored in the memory 302 to achieve various functions:

[0085] Obtain the business's data migration requirements when migrating heterogeneous databases;

[0086] Based on the above data migration requirements, configure a data migration strategy for the preset database middleware;

[0087] Create a data extraction process on the aforementioned database middleware, and obtain the target business data from the source database through the aforementioned data extraction process;

[0088] A data replication process is created on the aforementioned database middleware. Through this process, the target business data is sharded according to the aforementioned data migration strategy, and the sharded target business sub-data is migrated to the corresponding location in the target database.

[0089] In some embodiments, the above data migration strategy includes a data sharding mechanism and a number of data shards. The above-mentioned data sharding of the target business data according to the above data migration strategy includes:

[0090] The target business data is sharded according to the data sharding mechanism described above to obtain the target business sub-data of the above number of data shards.

[0091] The above-mentioned migration of each target business sub-data after sharding to the corresponding location in the target database includes:

[0092] Based on the above data migration strategy, the target number of business sub-business data in the above data shards will be migrated to the corresponding storage space in the target database.

[0093] In some embodiments, the target business data includes incremental business data and full business data, the data extraction process includes incremental extraction process and full extraction process, and obtaining the target business data from the source database includes:

[0094] The incremental extraction process described above retrieves the business logs corresponding to the business from the source database and parses and processes these business logs to obtain the incremental business data.

[0095] The full data of the business is obtained from the source database through the full extraction process described above.

[0096] In some embodiments, the message storage middleware corresponding to the database middleware described above includes an incremental data message set and a full data message set, and further includes:

[0097] Configure an incremental identifier for the above-mentioned incremental data message set, and configure a full identifier for the above-mentioned full data message set;

[0098] Based on the aforementioned incremental identifier, the aforementioned incremental business data is stored in the aforementioned incremental data message set;

[0099] Based on the aforementioned full identifier, the aforementioned full business data is stored in the aforementioned full data message set.

[0100] In some embodiments, the data replication process includes an incremental replication process and a full replication process. The process of partitioning the target business data according to the data migration strategy and migrating each partitioned sub-data item to its corresponding location in the target database includes:

[0101] The full business data is obtained from the full data message set through the full replication process described above. The full business data is then sharded according to the data migration strategy described above, and each sharded full business sub-data is migrated to the corresponding position in the target database described above.

[0102] The incremental business data is obtained from the incremental data message set through the incremental replication process, the incremental business data is sharded according to the data migration strategy, and the sharded incremental business sub-data is migrated to the corresponding position in the target database.

[0103] In some embodiments, the above-mentioned migration of each fragmented target service sub-data to the corresponding location in the target database includes:

[0104] Through the aforementioned data replication process, multiple concurrent threads are created based on the data volume of the target business data.

[0105] The sharded target business sub-data is allocated to each concurrent thread so that each concurrent thread can migrate the allocated target business sub-data to the corresponding location in the target database.

[0106] In some embodiments, it also includes:

[0107] During the migration of the sharded target business sub-data to the target database, the data in each field of the target business sub-data is processed accordingly based on the data mapping relationship corresponding to the field types in the target business sub-data.

[0108] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0109] Optional, such as Figure 3 As shown, the computer device 300 also includes: a touch screen display 303, a radio frequency circuit 304, an audio circuit 305, an input unit 306, and a power supply 307. The processor 301 is electrically connected to the touch screen display 303, the radio frequency circuit 304, the audio circuit 305, the input unit 306, and the power supply 307. Those skilled in the art will understand that... Figure 3 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0110] The touch display screen 303 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 303 may include a display panel and a touch panel. The display panel can be used to display messages input by the user or messages provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives the touch message from the touch detection device, converts it into touch point coordinates, and sends it to the processor 301. It can also receive and execute commands from the processor 301. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 301 to determine the type of touch event. Subsequently, the processor 301 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 303 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 303 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 303 can also be used as part of the input unit 306 to achieve input functions.

[0111] The radio frequency circuit 304 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other computer devices, and to transmit and receive signals with network devices or other computer devices.

[0112] Audio circuitry 305 can be used to provide an audio interface between a user and a computer device via a speaker and a microphone. Audio circuitry 305 converts received audio data into electrical signals, transmits them to the speaker, and the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 305, converted back into audio data, and output to processor 301 for processing. The audio data is then transmitted via radio frequency circuitry 304 to, for example, another computer device, or output to memory 302 for further processing. Audio circuitry 305 may also include an earphone jack to facilitate communication between peripheral headphones and the computer device.

[0113] The input unit 306 can be used to receive input numbers, character messages or user characteristic messages (such as fingerprints, iris, facial messages, etc.), and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0114] Power supply 307 is used to supply power to various components of computer device 300. Optionally, power supply 307 can be logically connected to processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 307 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0115] although Figure 3 As not shown in the diagram, computer equipment 300 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0117] As can be seen from the above, the computer device provided in this embodiment obtains the data migration requirements of the business when migrating to a heterogeneous database; based on the above data migration requirements, it configures a data migration strategy for a preset database middleware; it creates a data extraction process on the database middleware and obtains target business data from the source database through the data extraction process; it creates a data replication process on the database middleware and performs data sharding of the target business data according to the above data migration strategy through the data replication process, and migrates each sharded target business sub-data to the corresponding position in the target database.

[0118] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0119] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute steps in any of the data migration methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0120] Obtain the business's data migration requirements when migrating heterogeneous databases;

[0121] Based on the above data migration requirements, configure a data migration strategy for the preset database middleware;

[0122] Create a data extraction process on the aforementioned database middleware, and obtain the target business data from the source database through the aforementioned data extraction process;

[0123] A data replication process is created on the aforementioned database middleware. Through this process, the target business data is sharded according to the aforementioned data migration strategy, and the sharded target business sub-data is migrated to the corresponding location in the target database.

[0124] In some embodiments, the above data migration strategy includes a data sharding mechanism and a number of data shards. The above-mentioned data sharding of the target business data according to the above data migration strategy includes:

[0125] The target business data is sharded according to the data sharding mechanism described above to obtain the target business sub-data of the above number of data shards.

[0126] The above-mentioned migration of each target business sub-data after sharding to the corresponding location in the target database includes:

[0127] Based on the above data migration strategy, the target number of business sub-business data in the above data shards will be migrated to the corresponding storage space in the target database.

[0128] In some embodiments, the target business data includes incremental business data and full business data, the data extraction process includes incremental extraction process and full extraction process, and obtaining the target business data from the source database includes:

[0129] The incremental extraction process described above retrieves the business logs corresponding to the business from the source database and parses and processes these business logs to obtain the incremental business data.

[0130] The full data of the business is obtained from the source database through the full extraction process described above.

[0131] In some embodiments, the message storage middleware corresponding to the database middleware described above includes an incremental data message set and a full data message set, and further includes:

[0132] Configure an incremental identifier for the above-mentioned incremental data message set, and configure a full identifier for the above-mentioned full data message set;

[0133] Based on the aforementioned incremental identifier, the aforementioned incremental business data is stored in the aforementioned incremental data message set;

[0134] Based on the aforementioned full identifier, the aforementioned full business data is stored in the aforementioned full data message set.

[0135] In some embodiments, the data replication process includes an incremental replication process and a full replication process. The process of partitioning the target business data according to the data migration strategy and migrating each partitioned sub-data item to its corresponding location in the target database includes:

[0136] The full business data is obtained from the full data message set through the full replication process described above. The full business data is then sharded according to the data migration strategy described above, and each sharded full business sub-data is migrated to the corresponding position in the target database described above.

[0137] The incremental business data is obtained from the incremental data message set through the incremental replication process, the incremental business data is sharded according to the data migration strategy, and the sharded incremental business sub-data is migrated to the corresponding position in the target database.

[0138] In some embodiments, the above-mentioned migration of each fragmented target service sub-data to the corresponding location in the target database includes:

[0139] Through the aforementioned data replication process, multiple concurrent threads are created based on the data volume of the target business data.

[0140] The sharded target business sub-data is allocated to each concurrent thread so that each concurrent thread can migrate the allocated target business sub-data to the corresponding location in the target database.

[0141] In some embodiments, it also includes:

[0142] During the migration of the sharded target business sub-data to the target database, the data in each field of the target business sub-data is processed accordingly based on the data mapping relationship corresponding to the field types in the target business sub-data.

[0143] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0144] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0145] Since the computer program stored in the storage medium can execute the steps in any of the data migration methods provided in the embodiments of this application, the beneficial effects that any of the data migration methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0146] The data migration method, apparatus, computer equipment, and storage medium provided in the embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data migration method, characterized in that, The method includes: Obtain the business's data migration requirements when migrating heterogeneous databases; Based on the aforementioned data migration requirements, a data migration strategy is configured for the preset database middleware; A data extraction process is created on the database middleware, and the target business data is obtained from the source database through the data extraction process. A data replication process is created on the database middleware, and the target business data is sharded according to the data migration strategy through the data replication process, and the sharded target business sub-data is migrated to the corresponding position in the target database. The data migration strategy includes a data sharding mechanism and the number of data shards. The step of sharding the target business data according to the data migration strategy includes: The target business data is sharded according to the data sharding mechanism to obtain the target business sub-data of the specified number of data shards. Based on the data migration strategy, the target number of business sub-business data in the data shards are migrated to the corresponding storage space in the target database. The step of migrating the fragmented target service sub-data to the corresponding location in the target database includes: Through the data replication process, multiple concurrent threads are created based on the data volume of the target business data; The sharded target business sub-data is allocated to each concurrent thread so that each concurrent thread can migrate the allocated target business sub-data to the corresponding location in the target database.

2. The method according to claim 1, characterized in that, The target business data includes incremental business data and full business data; the data extraction process includes incremental extraction and full extraction processes; and obtaining the target business data from the source database includes: The incremental extraction process retrieves the business logs corresponding to the business from the source database and parses the business logs to obtain the incremental business data. The full extraction process retrieves the full business data corresponding to the business from the source database.

3. The method according to claim 2, characterized in that, The message storage middleware corresponding to the database middleware includes an incremental data message set and a full data message set, and also includes: Configure an incremental identifier for the incremental data message set, and configure a full identifier for the full data message set; Based on the incremental identifier, the incremental service data is stored in the incremental data message set; Based on the full identifier, the full amount of business data is stored in the full amount of data message set.

4. The method according to claim 3, characterized in that, The data replication process includes an incremental replication process and a full replication process. The step of using the data replication process to shard the target business data according to the data migration strategy, and then migrating each sharded target business sub-data to its corresponding location in the target database, includes: The full business data is obtained from the full data message set through the full replication process, the full business data is sharded according to the data migration strategy, and the sharded full business sub-data is migrated to the corresponding position in the target database. The incremental service data is obtained from the incremental data message set through the incremental replication process, the incremental service data is sharded according to the data migration strategy, and the sharded incremental service sub-data is migrated to the corresponding position in the target database.

5. The method according to any one of claims 1 to 4, characterized in that, Also includes: During the migration of the sharded target business sub-data to the target database, the data in each field of the target business sub-data is processed accordingly based on the data mapping relationship corresponding to the field types in the target business sub-data.

6. A data migration device, characterized in that, The device includes: The requirement elicitation module is used to obtain the data migration requirements of the business when migrating to heterogeneous databases. The strategy configuration module is used to configure a data migration strategy for a preset database middleware based on the data migration requirements. The data acquisition module is used to create a data extraction process on the database middleware and obtain target business data from the source database through the data extraction process. The data migration module is used to create a data replication process on the database middleware, and through the data replication process, to shard the target business data according to the data migration strategy, and to migrate each sharded target business sub-data to the corresponding position in the target database. The data migration strategy includes a data sharding mechanism and the number of data shards. The step of sharding the target business data according to the data migration strategy includes: The target business data is sharded according to the data sharding mechanism to obtain the target business sub-data of the specified number of data shards. Based on the data migration strategy, the target business sub-data of the specified number of data shards is migrated to the corresponding storage space in the target database. The step of migrating the fragmented target service sub-data to the corresponding location in the target database includes: Through the data replication process, multiple concurrent threads are created based on the data volume of the target business data; The sharded target business sub-data is allocated to each concurrent thread so that each concurrent thread can migrate the allocated target business sub-data to the corresponding location in the target database.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the program, implements the data migration method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores a plurality of instructions adapted for loading by a processor to execute the data migration method according to any one of claims 1 to 5.

Images