A method and system for lossless migration of multiple data sources during service migration

Abstract

This invention relates to a method and system for lossless migration of multiple data sources during service migration, comprising: creating a new database; copying the data table structure of OP short links in the original database to the new database; adding newly added OP short link data from the original database to the new database using an incremental insertion method according to preset conditions and preset intervals; splitting the OPs in the original database to obtain url-services; configuring the Nginx address of the url-services; configuring the OP short link domain name and the new domain name in the original database to the Nginx address of the OPs in the original database, and reverse proxying to the url-service address; modifying the DNS of the OP domain name in the original database, so that the DNS of the OP domain name in the original database points to both the Nginx address of the OPs in the original database and the Nginx address of the url-services; stopping the OP short link service in the original database; stopping the synchronization program, and the new service only connects to the newly created vertical database. During the migration process, the service is not terminated, the total data volume exceeds 100 million, and near real-time synchronization is achieved.

Description

Technical Field

[0001] This invention relates to the field of information system technology, and in particular to a method and system for lossless migration of multiple data sources during service migration. Background Technology

[0002] Data migration refers to the process of selecting, preparing, extracting, and transforming data, and permanently transferring it from one computer storage system to another. Furthermore, verifying the integrity of the migrated data and decommissioning the original old data storage are also considered part of the overall data migration process.

[0003] Data migration is a critical consideration for any system implementation, upgrade, or integration, and is typically performed in a manner that is as automated as possible to free up human resources from tedious tasks. Data migration can occur for a variety of reasons, including server or storage device replacement, maintenance or upgrades, application migration, website integration, disaster recovery, and data center migration.

[0004] Physical migration is suitable for large-scale migrations involving large amounts of data. It includes two methods: copying data files and using the XtraBackup backup tool. Physical migration can be used between different servers. On the new server, install the same version of the database software, create the same directories, and ideally, ensure the configuration files are identical to the original database. Then, copy the data files and log files from the original database, configure the file group permissions, and finally start the database on the new server using the `mysqld` command.

[0005] The advantage of using a physical migration solution is that it is relatively fast, but it requires downtime for migration and the MySQL version and configuration must be the same as the original server, which may also cause unknown problems.

[0006] In comparison, logical migration has a wider range of applications; it can be used for both partial and full migrations. The most common method for logical migration is to export and then import data using backup tools like mysqldump.

[0007] Lossless migration refers to a process where incremental logs are recorded at the start of the data migration without downtime, and then processed after the migration is complete. Finally, writes to the data to be migrated can be paused (application shutdown) to ensure all incremental data has been processed before switching the data source, resuming all writes, and completing the migration.

[0008] The current mainstream solutions include: point-to-point downtime migration, MySQL binlog solution, trigger solution, and middleware solution.

[0009] The advantage of using a physical migration solution is that it is relatively fast, but it requires downtime for migration and the MySQL version and configuration must be the same as the original server, which may also cause unknown problems.

[0010] Targeted migration requires system downtime and service suspension. As the amount of data increases, the downtime will lengthen. For data in the hundreds of millions, it may take several days. Businesses with such large data volumes generally cannot tolerate this level of downtime for maintenance.

[0011] The Binlog migration solution parses binlog logs, performs special processing on each row of data according to business needs, and writes the data to the new database. The migration process does not require downtime. When the data migration is basically complete, the application is stopped, and the migration is fully completed before switching the application to the new database. The downtime is very short, requiring only 1-2 minutes or less, which is tolerable for non-financial businesses.

[0012] The trigger-based approach involves backing up the old MySQL table structure to a new MySQL database. In the new database, new table structures are created, and modifications are made to the old tables. Triggers are then created to simultaneously write data to the new MySQL tables. The old MySQL data is dumped and imported into the new MySQL database; the new tables should now contain the corresponding data. Master-slave replication is then enabled to ensure data consistency with the master database. The application is then switched to this migration solution. Downtime is extremely short, requiring only 1-2 minutes or less.

[0013] Middleware solutions typically employ intermediate storage devices to cache data, achieving near real-time data synchronization, but their development and design are relatively complex.

[0014] Therefore, existing technologies cannot simultaneously satisfy multiple data sources in both horizontal and vertical databases, leading to data migration issues in some data tables. Summary of the Invention

[0015] To address the shortcomings of existing technologies, this invention proposes a method and system for lossless migration of multiple data sources during service migration. During the migration process, the service is not terminated, and some data from the horizontal and vertical databases are migrated to the new vertical database simultaneously. The total data volume exceeds 100 million, and the data is synchronized in near real-time.

[0016] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0017] A method for lossless migration of multiple data sources during service migration, characterized by comprising:

[0018] S1. Create a new database;

[0019] S2. Copy the data table structure of the OP short links in the original database to the newly created database; synchronize the OP short link data in the original database to the newly created database;

[0020] S3. According to preset conditions and preset intervals, add the short link data newly added by OP in the original database to the newly created database using the incremental insertion method;

[0021] S4. Split the OPs in the original database to obtain the url-service;

[0022] S5. Configure the Nginx address for url-service;

[0023] S6. Configure the OP short link domain name in the original database and the new domain name to the Nginx address of the OP in the original database, and reverse proxy to the url-service address;

[0024] S7. Modify the DNS of the OP domain name in the original database, so that the DNS of the OP domain name in the original database points to both the Nginx address of the OP in the original database and the Nginx address of the url-service.

[0025] S8. Stop the OP short link service in the original database;

[0026] S9. Stop the synchronization process; the new service will only connect to the newly created vertical database.

[0027] Furthermore, the short chain data includes: short chain configuration data and short chain record data.

[0028] Furthermore, the preset interval time is 10 seconds, and the preset condition is that no more than 2,000 short link data are retrieved each time.

[0029] Further, step S3 includes:

[0030] S31. Retrieve the last synchronized data record in the newly created vertical database and set the create_time value of the last synchronized data record to the offset value.

[0031] S32. Retrieve data from the original database whose create_time value is greater than the offset value, and name it data1;

[0032] S33. Retrieve data from the newly created vertical database whose create_time value is greater than the offset value, and name it data2;

[0033] S34. Compare data1 and data2 to identify duplicate data;

[0034] S35. Delete duplicate data and add the remaining data to the newly created database.

[0035] Furthermore, in step S1, the database is a vertical database.

[0036] Furthermore, the url-service is a short-connection service split from the OP in the original database. It has three data sources and adopts a dual-write mode. Data is read first from the newly created database. If the data cannot be read from the newly created database, it is read from the original database.

[0037] This invention also relates to a system for lossless migration of multiple data sources during service migration, characterized in that it includes:

[0038] The database creation module is used to create new databases.

[0039] The data replication module is used to copy the data table structure of the OP short chain in the original database to the newly created database;

[0040] The data addition module is used to add short link data newly added by OP in the original database to the newly created database according to preset conditions and preset intervals using an incremental insertion method.

[0041] The splitting module is used to split the OPs in the original database to obtain url-services;

[0042] The configuration module is used to configure the Nginx address of url-service;

[0043] Configure the OP short link domain name in the original database and the new domain name to the Nginx address of the OP in the original database, and reverse proxy to the url-service address;

[0044] Modify the DNS of the OP domain name in the original database so that the DNS of the OP domain name in the original database points to both the Nginx address of the OP in the original database and the Nginx address of the url-service.

[0045] The service stop module is used to stop the OP short-link service in the original database; stop the synchronization program, and the new service will only connect to the newly created vertical database.

[0046] The present invention also relates to a computer-readable storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method for lossless migration of multiple data sources during service migration.

[0047] The present invention also relates to an electronic device, characterized in that it includes a processor and a memory;

[0048] The memory is used to store the original database and the newly created database;

[0049] The processor is used to execute the above-described method for lossless migration of multiple data sources during service migration by calling the original database and the newly created database.

[0050] The present invention also relates to a computer program product, including a computer program and / or instructions, characterized in that, when the computer program and / or instructions are executed by a processor, they implement the steps of the above-described method for lossless migration of multiple data sources during service migration.

[0051] The beneficial effects of this invention are as follows:

[0052] This invention presents a method and system for lossless migration of multiple data sources during service migration. It involves lossless migration schemes for various data sources, including horizontal and vertical databases, supporting partial data migration from multiple data sources in both horizontal and vertical databases. During data migration, services continue uninterrupted, achieving seamless migration. Data in the source and new databases is synchronized in near real-time. Incremental data migration avoids large-scale data I / O and does not utilize other middleware. The design is relatively simple, with low development costs, and strong operability and security. During the migration process, services remain uninterrupted, simultaneously migrating partial data from both horizontal and vertical databases to the new vertical database, with a total data volume exceeding 100 million records and near real-time synchronization. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of a method for lossless migration of multiple data sources during service migration according to the present invention.

[0054] Figure 2 This is a schematic diagram of the system structure for lossless migration of multiple data sources during service migration according to the present invention. Detailed Implementation

[0055] Data migration refers to the process of selecting, preparing, extracting, and transforming data, and permanently transferring it from one computer storage system to another. Furthermore, verifying the integrity of the migrated data and decommissioning the original old data storage are also considered part of the overall data migration process.

[0056] Data migration is a critical consideration for any system implementation, upgrade, or integration, and is typically performed in a manner that is as automated as possible to free up human resources from tedious tasks. Data migration can occur for a variety of reasons, including server or storage device replacement, maintenance or upgrades, application migration, website integration, disaster recovery, and data center migration.

[0057] To better understand the content of this invention, a detailed description will be provided in conjunction with the accompanying drawings and embodiments.

[0058] Vertical and Horizontal Databases: Common database sharding methods include horizontal and vertical databases. Vertical databases separate data into multiple databases based on business dimensions and data access volume. For example, common configuration information might be stored in one database for separate maintenance. Horizontal databases, as the name suggests, divide the data of a single table into different databases, with both databases having identical table structures.

[0059] OP: Original service system name;

[0060] url-service: The name of the new service split from the original service;

[0061] create_time: The time when the data was created;

[0062] offset: data offset;

[0063] Short link: The name of the data generated by the service;

[0064] DNS: DNS stands for Domain Name System. To understand it, we first need to understand what a domain name is. A domain name (English: DomainName), also known as a network domain, is a string of names separated by dots on the Internet, used to locate and identify a computer (sometimes also referring to its geographical location) during data transmission. Because IP addresses are inconvenient to remember and cannot display the name and nature of the address organization, domain names were designed, and the Domain Name System (DNS) is used to map domain names to IP addresses, making it easier for people to access the Internet without having to remember the strings of IP addresses that can be directly read by machines.

[0065] Nginx address: Nginx (engine x) is a high-performance HTTP and reverse proxy web server. The Nginx address refers to the IP address of the Nginx server machine.

[0066] The first aspect of this invention relates to a method for lossless migration of multiple data sources during service migration, the steps of which are as follows: Figure 1 The method shown includes:

[0067] Create a new vertical database instance, copy the data table structure of the OP short chain, and synchronize the relevant data of the original short chain to the new instance.

[0068] S1. Create a new database; this database is a vertical database.

[0069] S2. Copy the data table structure of the OP short links in the original database to the newly created database; synchronize the OP short link data in the original database to the newly created database;

[0070] The short chain data includes: short chain configuration data and short chain record data.

[0071] Start the synchronization service. The synchronization service is responsible for pulling the newly added short link configuration and short link record data from the OP database every 10 seconds. Each pull will not exceed 2000 records and will be performed incrementally.

[0072] Incremental Insertion: Retrieves the last synchronized record from the newly created instance, and sets the create_time of the last record as the offset. Data with create_time > offset from the OP database is recorded as data1, with no more than 2000 records. Data with create_time > offset from the new instance is recorded as data2. The retrieved data are compared, duplicate data is removed, and the data is inserted into the new database instance.

[0073] S3. According to preset conditions and preset intervals, add the short link data newly added by OP in the original database to the newly created database using the incremental insertion method;

[0074] The preset interval is 10 seconds, and the preset condition is that no more than 2,000 short link data are retrieved each time.

[0075] Step S3 includes:

[0076] S31. Retrieve the last synchronized data record in the newly created vertical database and set the create_time value of the last synchronized data record to the offset value.

[0077] S32. Retrieve data from the original database whose create_time value is greater than the offset value, and name it data1;

[0078] S33. Retrieve data from the newly created vertical database whose create_time value is greater than the offset value, and name it data2;

[0079] S34. Compare data1 and data2 to identify duplicate data;

[0080] S35. Delete duplicate data and add the remaining data to the newly created database.

[0081] Start the url-service service. url-service is a split short-connection service with three data sources, employing a dual-write approach. Data is read first from the new database instance; if not found there, it is read from the OP database. After starting the url-service service, configure the OP short-connection domain and the new domain to be reverse proxied to the url-service address in Nginx. Modify the DNS of the OP domain to point to both the original OP Nginx address and the new service's Nginx address.

[0082] S4. Split the OPs in the original database to obtain the url-service;

[0083] The url-service is a short-connection service split from the OP in the original database. It has three data sources and adopts a dual-write mode. Data is read first from the newly created database. If the data cannot be read from the newly created database, it is read from the original database.

[0084] Configure the nginx for url-service to listen to the original OP short link domain. Modify the DNS so that the original OP short link domain points to the nginx server address of url-service. Stop the synchronization program. Release the url-service. url-service will only connect to the new database instance. OPs can listen to the OP short link related services as needed.

[0085] S5. Configure the Nginx address for url-service;

[0086] S6. Configure the OP short link domain name in the original database and the new domain name to the Nginx address of the OP in the original database, and reverse proxy to the url-service address;

[0087] S7. Modify the DNS of the OP domain name in the original database, so that the DNS of the OP domain name in the original database points to both the Nginx address of the OP in the original database and the Nginx address of the url-service.

[0088] In the OP release, the short link management service is disabled, the gateway interface for generating short links is removed, and the HTTP short link redirection service is retained.

[0089] S8. Stop the OP short link service in the original database;

[0090] S9. Stop the synchronization process; the new service will only connect to the newly created vertical database.

[0091] This migration will require a total of three releases: the first, second, and third steps constitute one release (url-service release), the fourth step is the OP release, and the fifth step is the url-service release. During the migration, business operations will remain uninterrupted and seamless, and the entire migration is expected to be completed in two weeks.

[0092] Another aspect of the present invention relates to a system for lossless migration of multiple data sources during service migration, the structure of which is as follows: Figure 2 As shown, it includes:

[0093] The database creation module is used to create new databases.

[0094] The data replication module is used to copy the data table structure of the OP short chain in the original database to the newly created database;

[0095] The data addition module is used to add short link data newly added by OP in the original database to the newly created database according to preset conditions and preset intervals using an incremental insertion method.

[0096] The splitting module is used to split the OPs in the original database to obtain url-services;

[0097] The configuration module is used to configure the Nginx address of url-service;

[0098] Configure the OP short link domain name in the original database and the new domain name to the Nginx address of the OP in the original database, and reverse proxy to the url-service address;

[0099] Modify the DNS of the OP domain name in the original database so that the DNS of the OP domain name in the original database points to both the Nginx address of the OP in the original database and the Nginx address of the url-service.

[0100] The service stop module is used to stop the OP short-link service in the original database; stop the synchronization program, and the new service will only connect to the newly created vertical database.

[0101] By using this system, the aforementioned computational processing methods can be executed and the corresponding technical effects can be achieved.

[0102] Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps of the methods in the above embodiments, wherein the computer-readable storage medium stores a computer program that, when executed by a processor, implements all the steps of the methods in the above embodiments.

[0103] Embodiments of the present invention also provide an electronic device for performing the above-described method. As an implementation device for the method, the electronic device has at least a processor and a memory. In particular, the memory stores data and related computer programs required for performing the method, such as an original database and a newly created database. The processor calls the data and programs in the memory to execute all the steps of the method and obtain the corresponding technical effect.

[0104] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges. The bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor during operation.

[0105] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) used may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a buffer, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, which can store the aforementioned data information, and may also store programs for executing the information, and the processor can execute the program stored in the memory to achieve information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it can be a button or touch input device; the power supply is used to provide power to the electronic device; the display is used to display images and text, for example, it can be an LCD display. The communication module is a transmitter / receiver that transmits and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which can be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules can be incorporated into the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) is also coupled to a speaker and microphone via an audio processor to provide audio output through the speaker and receive audio input from the microphone, thereby enabling typical telecommunications functions. The audio processor can include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor is coupled to a central processing unit, enabling on-device recording via the microphone and on-device playback of stored sound via the speaker.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0110] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for lossless migration of multiple data sources during service migration, characterized in that, include: S1. Create a new database, which is a vertical database; S2. Copy the data table structure of the OP short links in the original database to the newly created database; synchronize the OP short link data in the original database to the newly created database; S3. According to preset conditions and preset intervals, add the short link data newly added by OP in the original database to the newly created database using the incremental insertion method; S4. Split the OPs in the original database to obtain the url-service; S5. Configure the Nginx address for url-service; S6. Configure the OP short link domain name in the original database and the new domain name to the Nginx address of the OP in the original database, and reverse proxy to the url-service address; S7. Modify the DNS of the OP domain name in the original database, so that the DNS of the OP domain name in the original database points to both the Nginx address of the OP in the original database and the Nginx address of the url-service. S8. Stop the OP short link service in the original database; S9. Stop the synchronization process; the new service will only connect to the newly created vertical database. The url-service is a short-connection service split from the OP in the original database. It has three data sources and adopts a dual-write mode. Data is read first from the newly created database. If the data cannot be read from the newly created database, it is read from the original database.

2. The method as described in claim 1, characterized in that, The short chain data includes: short chain configuration data and short chain record data.

3. The method as described in claim 1, characterized in that, The preset interval is 10 seconds, and the preset condition is that no more than 2,000 short link data are retrieved each time.

4. The method as described in claim 1, characterized in that, Step S3 includes: S31. Retrieve the last synchronized data record in the newly created vertical database and set the create_time value of the last synchronized data record to the offset value. S32. Retrieve data from the original database whose create_time value is greater than the offset value, and name it data1; S33. Retrieve data from the newly created vertical database whose create_time value is greater than the offset value, and name it data2; S34. Compare data1 and data2 to identify duplicate data; S35. Delete duplicate data and add the remaining data to the newly created database.

5. A system for lossless migration of multiple data sources during service migration, characterized in that, include: The database creation module is used to create new databases, which are vertical databases. The data replication module is used to copy the data table structure of the OP short chain in the original database to the newly created database; The data addition module is used to add short link data newly added by OP in the original database to the newly created database according to preset conditions and preset intervals using an incremental insertion method. The splitting module is used to split the OPs in the original database to obtain url-services; The configuration module is used to configure the Nginx address of url-service; Configure the OP short link domain name in the original database and the new domain name to the Nginx address of the OP in the original database, and reverse proxy to the url-service address; Modify the DNS of the OP domain name in the original database so that the DNS of the OP domain name in the original database points to both the Nginx address of the OP in the original database and the Nginx address of the url-service. The service stop module is used to stop the OP short link service in the original database; stop the synchronization program, and the new service will only connect to the newly created vertical database; The system also includes a url-service which is a short-connection service split from the OP in the original database. It has three data sources and adopts a dual-write method. Data is read first from the newly created database. If the data cannot be read from the newly created database, it is read from the original database.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements a method for lossless migration of multiple data sources during service migration, as described in any one of claims 1 to 4.

7. An electronic device, characterized in that, Including processor and memory; The memory is used to store the original database and the newly created database; The processor is configured to execute, by invoking the original database and the newly created database, a method for lossless migration of multiple data sources during service migration as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by a processor, they implement the steps of the method for lossless migration of multiple data sources during service migration as described in any one of claims 1 to 4.

Images