Method for smoothly migrating mass data based on asynchronous queue mode

Abstract

The invention discloses a method for smoothly migrating mass data based on an asynchronous queue mode. The method comprises the steps that S1, 16 tables are newly built in a main library; s2, establishing and adding a synchronization task monitoring original table database operation binary log; s3, exporting total data of the original table to generate a data file; s4, splitting the program into 16 files; s5, recharging the 16 split files into 16 tables of a backup library node; s6, importing a backup node table space into a tool library node; s7, importing a tool library node table space into a slave library node; s8, using the completed nodes to import the table space into the main library nodes; s9, starting a consumption program in the consumption queue to carry out historical message data consumption; by adopting the migration scheme, the data migration efficiency can be improved; by adopting the migration scheme, if an uncontrollable problem is encountered in the migration process, quick rollback can be carried out without online code modification of a task, so that the stability of an online system is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of online car-hailing platforms, and specifically relates to a method for smoothly migrating massive data based on asynchronous queues. Background Art

[0002] The online ride-hailing driver order-taking platform pushes more and more data as the company develops. The original form of split tables and single tables has reached hundreds of millions. Under ideal conditions, the recommended single database table is 20 million, which is several times the amount. The following problems will arise:

[0003] The increased pressure on a single database table causes data query and writing to cause higher pressure on the service CPU and disk; the data loading speed seen by users on their mobile phones is slow, resulting in a poor user experience.

[0004] In order to solve the risk of the above problem, data archiving is usually used. However, due to business restrictions, historical data of more than half a year must be retained. Currently, the table can only be split based on the current data volume to disperse the query volume of data;

[0005] If you ask the database administrator to help migrate directly by importing the table, you can only import the data in a point-to-point manner. This makes it impossible to perform sharding calculations on the data, resulting in the new table and the old table having the same amount of data, which requires a program to delete. At the same time, there is a problem of incremental data between the new and old tables. This is not conducive to quickly and smoothly migrating data to the new table for immediate use.

[0006] To this end, the present invention designs a method for smoothly migrating massive data based on an asynchronous queue. Summary of the invention

[0007] The purpose of the present invention is to provide a method for smoothly migrating massive data based on an asynchronous queue to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a method for smoothly migrating massive data based on an asynchronous queue, S1: creating 16 tables in the main database;

[0009] S2: Create and add a synchronization task to monitor the binary log of the original table database operation;

[0010] S3: Export the full data of the original table to generate a data file;

[0011] S4: The program splits 16 files;

[0012] S5: The 16 split files are returned to the 16 tables of the backup database node;

[0013] S6: Import the backup node tablespace into the tool library node;

[0014] S7: Import the tool library node tablespace to the slave library node;

[0015] S8: Use the completed node to import the tablespace into the master database node;

[0016] S9: Start the consumer program in the consumer queue to consume historical message data;

[0017] Specific interface migration process:

[0018] A1: The control layer is responsible for providing data entry;

[0019] A2: The program interceptor parses the sharding keyword based on the parameters passed in by the control layer;

[0020] A3: The configuration center stores synchronization rules;

[0021] A4: Calculate the synchronization rule results based on the sharding keywords parsed by the program interceptor and the synchronization rules of the configuration center and store them in the local cache.

[0022] A5: The database persistence framework interceptor parses the parameters in the database operation statement. If the sharding keyword is parsed, it goes to the local cache and obtains the corresponding rules to determine whether to use the new table or the historical table. If it is a new table, the rule data is written to the new table; if it is an old table, the rule data is written to the old table.

[0023] A6: The database persistence framework interceptor parses the parameters of the database operation language. If the sharding keyword cannot be parsed, it is considered that the sharding keyword is missing. At the same time, monitoring is performed to provide developers with problem analysis and supplementation of sharding keywords.

[0024] Preferably, the synchronization rules mainly include:

[0025] a) City switching rules: Calculate the city based on the shard keyword and then go to the configuration center to get the rules based on the obtained city to determine whether to use the new table or the historical table;

[0026] b) Percentage switching rule: The value obtained by modulo calculation based on the sharding keyword is then used according to the configuration center's rules to determine whether to use the new table or the historical table;

[0027] c) Specify the shard keyword switching rule: Determine whether the configuration center contains the shard to be switched based on the shard keyword to determine whether to use the new table or the historical table.

[0028] Preferably, in step S2, the binary log records all data change operations of the database, including data insertion, update, deletion and database structure change.

[0029] Preferably, in step A1, the original interface form is kept unchanged to ensure that the upstream system cannot be changed.

[0030] Compared with the prior art, the beneficial effects of the present invention are: the present invention can improve the efficiency of data migration by adopting the migration solution; if an uncontrollable problem is encountered during the migration process, the migration solution can be quickly rolled back without the need to modify the code of the task online to ensure the stability of the online system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of a specific data migration process of the present invention.

[0032] Figure 2 It is a schematic diagram of the program execution flow of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] See also Figure 1-2 The present invention provides a technical solution: a method for smoothly migrating massive data based on an asynchronous queue. The specific data migration process is as follows:

[0035] S1: Create 16 new tables in the main database. Here, we assume that the capacity of the new tables needs to be expanded by 16 tables.

[0036] S2: Create and add synchronization tasks to monitor the binary log of database operations on the original table, including data insertion, update, deletion, and database structure changes;

[0037] S3: Export the full data of the original table to generate a data file;

[0038] S4: The program splits 16 files;

[0039] S5: The 16 split files are returned to the 16 tables of the backup database node;

[0040] S6: Import the backup node tablespace into the tool library node;

[0041] S7: Import the tool library node tablespace to the slave library node;

[0042] S8: Use the completed node to import the tablespace into the master database node;

[0043] S9: Start the consumer program in the consumer queue to consume historical message data.

[0044] Specific interface migration process:

[0045] A1: The control layer is responsible for providing data entry. The original interface format remains unchanged here to ensure that the upstream system cannot be changed.

[0046] A2: The program interceptor parses the sharding keyword based on the parameters passed in by the control layer.

[0047] A3: The configuration center stores synchronization rules.

[0048] Synchronization rules mainly include:

[0049] a) City switching rules: Calculate the city based on the shard keyword and then go to the configuration center to get the rules based on the obtained city to determine whether to use the new table or the historical table.

[0050] b) Percentage switching rule: The value obtained by modulo calculation based on the sharding keyword is then used according to the configuration center's rules to determine whether to use the new table or the historical table.

[0051] c) Specify the shard keyword switching rule: Determine whether the configuration center contains the shard to be switched based on the shard keyword to determine whether to use the new table or the historical table.

[0052] A4: The synchronization rule results are calculated based on the sharding keywords parsed by the program interceptor and the synchronization rules of the configuration center and stored in the local cache.

[0053] A5: The database persistence framework interceptor parses the parameters in the database operation statement. If the sharding keyword is parsed, it goes to the local cache to obtain the corresponding rules to determine whether to use the new table or the historical table. If it is a new table, the rule data is written to the new table; if it is an old table, the rule data is written to the old table.

[0054] A6: The database persistence framework interceptor parses the parameters of the database operation language. If the sharding keyword cannot be parsed, it is considered that the sharding keyword is missing. At the same time, monitoring is performed to provide developers with problem analysis and supplementation of sharding keywords.

[0055] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for smoothly migrating massive data based on an asynchronous queue, characterized by: S1: 16 new tables are created in the main database; S2: Create and add a synchronization task to monitor the binary log of the original table database operation; S3: Export the full data of the original table to generate a data file; S4: The program splits 16 files; S5: The 16 split files are returned to the 16 tables of the backup database node; S6: Import the backup node tablespace into the tool library node; S7: Import the tool library node tablespace to the slave library node; S8: Use the completed node to import the tablespace into the master database node; S9: Start the consumer program in the consumer queue to consume historical message data; Specific interface migration process: A1: The control layer is responsible for providing data entry; A2: The program interceptor parses the sharding keyword based on the parameters passed in by the control layer; A3: The configuration center stores synchronization rules; A4: Calculate the synchronization rule results based on the sharding keywords parsed by the program interceptor and the synchronization rules of the configuration center and store them in the local cache. A5: The database persistence framework interceptor parses the parameters in the database operation statement. If a sharding keyword is parsed, it goes to the local cache and obtains the corresponding rules to determine whether to use the new table or the historical table. If it is a new table rule, the data is written to the new table; if it is an old table rule, the data is written to the old table; A6: The database persistence framework interceptor parses the parameters of the database operation language. If the sharding keyword cannot be parsed, it is considered that the sharding keyword is missing. At the same time, monitoring is performed to provide developers with problem analysis and supplementation of sharding keywords.

2. The method for smoothly migrating massive data based on asynchronous queues according to claim 1 is characterized in that: Synchronization rules mainly include: a) City switching rules: Calculate the city based on the shard keyword and then go to the configuration center to get the rules based on the obtained city to determine whether to use the new table or the historical table; b) Percentage switching rule: The value obtained by modulo calculation based on the sharding keyword is then used according to the configuration center's rules to determine whether to use the new table or the historical table; c) Specify the shard keyword switching rule: Determine whether the configuration center contains the shard to be switched based on the shard keyword to determine whether to use the new table or the historical table.

3. The method for smoothly migrating massive data based on asynchronous queues according to claim 1 is characterized in that: In step S2, the binary log records all data change operations of the database, including data insertion, update, deletion, and database structure changes.

4. The method for smoothly migrating massive data based on asynchronous queues according to claim 1 is characterized in that: In step A1, keep the original interface form unchanged to ensure that the upstream system cannot be changed.

Images