A real-time synchronization system and method for massive data

Abstract

The present invention discloses a real-time synchronization system and method for massive data, wherein the method comprises: using Canal to obtain the incremental log of the source database, generating incremental data according to the incremental log, and pushing the incremental data as a message record to the message queue; using the incremental synchronization system to obtain the message record in the message queue, distributing the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule; using the incremental synchronization system to process the message record in the message subqueue and then storing it in the target database. The present invention has high decoupling and can ensure the real-time and consistency of massive data synchronization.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a real-time synchronization system and method for massive data. Background Art

[0002] With the iteration of Internet technology and the surge in the number of users, the platform can no longer meet daily use and needs to be reconstructed. If the reconstruction involves database transformation, the original platform data needs to be fully migrated and incrementally migrated, and database reconstruction may involve table adjustments, field types, etc. Therefore, when migrating data, the table cannot be copied directly, and operations such as conversion and splitting need to be performed based on the data.

[0003] In order to ensure the stability of the platform, incremental migration is usually used. The existing common incremental migration solutions are: scheduled task mode, traffic replication mode and tracking mode. Incremental migration requires real-time data and platform decoupling. When the amount of incremental data is very large, it is impossible to achieve real-time synchronization of a large amount of data. The above solutions do not meet the requirements of platform incremental migration. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time synchronization system and method for massive data, so as to solve the technical problem that the existing technology cannot meet the real-time and decoupling requirements during the incremental migration of large amounts of data.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A real-time synchronization system for massive data, comprising:

[0007] Source database, Canal cluster, message queue cluster, incremental synchronization system, target database;

[0008] Wherein, the source database generates incremental logs;

[0009] The Canal cluster monitors the source database and obtains incremental logs, generates incremental data according to the incremental logs, and pushes the incremental data as message records to the message queue cluster;

[0010] The message queue cluster receives the message record pushed by the Canal cluster;

[0011] The incremental synchronization system monitors the message queue cluster and obtains message records, distributes the message records to corresponding message subqueues for concurrent consumption according to preset distribution rules, processes the message records in the message subqueues and stores them in the target database;

[0012] The target database stores the message records processed by the incremental synchronization system.

[0013] The present invention also provides a real-time synchronization method for massive data, which is applied to a real-time synchronization system for massive data. The method comprises:

[0014] Using Canal to obtain the incremental log of the source database, generating incremental data according to the incremental log, and pushing the incremental data as a message record to the message queue cluster;

[0015] Using the incremental synchronization system to obtain the message records in the message queue, and distributing the message records to the corresponding message subqueues for concurrent consumption according to the preset distribution rules;

[0016] The message records in the message subqueue are processed by using an incremental synchronization system and then stored in a target database.

[0017] Optionally, the preset distribution rule includes:

[0018] Pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules;

[0019] Among them, the pre- and post-table association rule distributes message records according to the table of current incremental synchronization, the table field value association rule distributes message records according to the fields in the table, and the single-table concurrent consumption rule distributes message records in parallel according to the modulus of the fields in a single table.

[0020] Optionally, it includes: using at least one of the preceding and following table association rules, the table field value association rules and the single table concurrent consumption rules according to the priority;

[0021] Among them, the priorities of the table field value association rule, the pre- and post-table association rule, and the single-table concurrent consumption rule are from high to low.

[0022] Optionally, after acquiring the message record in the message queue by using the incremental synchronization system, the method further includes:

[0023] If the message record is an update operation, it will enter the filtering module for verification.

[0024] Optionally, the filter module verification includes:

[0025] S11: Determine whether a message filtering rule is configured. If so, execute S12. If not, execute the step of "distributing the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule";

[0026] S12: Verify whether the message record needs to be filtered. If so, end the operation. If not, execute the step of "distributing the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule".

[0027] Optionally, it also includes:

[0028] Determine whether the message record is abnormal based on the preset abnormal conditions. If abnormal, add the abnormal message record to the retry queue and perform a retry operation.

[0029] Optionally, after adding the abnormal message record to the retry queue and performing the retry operation, the following steps are also included:

[0030] When the number of retry operations reaches the preset threshold, the abnormal message record is processed for abnormal storage operation.

[0031] Optionally, before using the Canal cluster to obtain the incremental log of the source database, the following steps are also included:

[0032] Deploy the Canal cluster and the message queue, and configure the tables to be synchronized and the address of the message queue to be pushed through the Canal configuration file.

[0033] Optionally, before using the incremental synchronization system to obtain the message record in the message queue, the method further includes:

[0034] Deploy the incremental synchronization system and configure preset distribution rules.

[0035] The present invention provides a real-time synchronization system and method for massive data, wherein the method comprises: using Canal to obtain incremental logs of a source database, generating incremental data according to the incremental logs, and pushing the incremental data as message records to a message queue; using an incremental synchronization system to obtain message records in the message queue, and distributing the message records to corresponding message subqueues for concurrent consumption according to preset distribution rules; and using the incremental synchronization system to process the message records in the message subqueues and store them in a target database.

[0036] In view of this, the beneficial effects brought by the present invention are:

[0037] The present invention utilizes Canal to monitor the source database and obtain incremental logs, generates incremental data according to the incremental logs and pushes them to the message queue, focuses on the data, does not need to pay attention to the upper-level business applications, avoids association with the business, and has high decoupling; according to the configured preset distribution rules, the incremental synchronization system distributes the message records to specific message sub-queues, flexibly configures data consumption, can speed up message consumption, and ensure the real-time synchronization of large amounts of data; through the retry mechanism and the abnormal reminder mechanism, the consistency of the incremental synchronization data is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural schematic diagram of the system device of the present invention;

[0039] Figure 2 The process of the embodiment of the method of the present invention is shown in FIG. Figure 1 ;

[0040] Figure 3 The process of the embodiment of the method of the present invention is shown in FIG. Figure 2 ;

[0041] Figure 4 A schematic diagram of a pre- and post-table association rule flow chart of an embodiment of the method of the present invention;

[0042] Figure 5 A schematic diagram of a table field value association rule flow chart of an embodiment of the method of the present invention;

[0043] Figure 6 A schematic diagram of a single-table concurrent consumption rule flow chart of an embodiment of the method of the present invention;

[0044] Figure 7 It is a flowchart of the method of the present invention including filtering and abnormal retry;

[0045] Figure 8 The figure is a flow chart of another embodiment of the method of the present invention. DETAILED DESCRIPTION

[0046] The embodiments of the present invention provide a system and method for real-time synchronization of massive data to solve the technical problem that the prior art cannot meet the real-time and decoupling requirements during the incremental migration of a large amount of data.

[0047] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0049] The present invention is mainly aimed at incremental migration and real-time synchronization of data from third-party systems; incremental migration is when new, modified, and deleted data is generated on the old platform during full migration, so this series of data needs to be incrementally synchronized. In order to ensure the stability of the platform, the new platform launch plan generally does not adopt direct cutting, but generally adopts methods such as gradual volume expansion, and only performs full cutting after stable operation. During this period, the data generated by the old platform also needs to be synchronized to the new platform in real time to ensure the real-time and validity of the data.

[0050] To achieve incremental migration, the commonly used technical solutions currently include:

[0051] (1) Scheduled task mode: obtain data through database synchronization or interface; obtain data regularly according to time parameters and synchronize it to the new platform. This method is not very real-time and performance bottlenecks must be considered.

[0052] (2) Traffic replication mode: This method replicates the traffic of the old platform and adds the same interface to the new platform to process data. Although this method can ensure the real-time nature of the data, it requires a very close familiarity with the business processing of the old platform to ensure that the data processed by the entire business request is consistent with that of the old platform.

[0053] (3) Tracking mode: This mode notifies the new platform of data operation by calling interfaces or other means when the underlying data is operated, and the new platform then processes the data. This mode requires the transformation of the old platform.

[0054] Incremental migration requires real-time data and platform decoupling. When the amount of incremental data is very large, the above solutions do not meet the requirements of platform incremental migration. There are also solutions based on Canal synchronization in the prior art, but in order to ensure the order of data consumption, it is impossible to achieve real-time synchronization of large amounts of data.

[0055] See also Figure 1 The present invention provides an embodiment of a real-time synchronization system for massive data, including:

[0056] Source database, Canal cluster, message queue cluster, incremental synchronization system, target database;

[0057] Wherein, the source database generates incremental logs;

[0058] The Canal cluster monitors the source database and obtains incremental logs, generates incremental data according to the incremental logs, and pushes the incremental data as message records to the message queue cluster;

[0059] The message queue cluster receives the message record pushed by the Canal cluster;

[0060] The incremental synchronization system monitors the message queue cluster and obtains message records, distributes the message records to corresponding message subqueues for concurrent consumption according to preset distribution rules, processes the message records in the message subqueues and stores them in the target database;

[0061] The target database stores the message records processed by the incremental synchronization system.

[0062] The real-time synchronization method of mass data provided by the present invention realizes the real-time synchronization of mass data based on Canal and message queue, and includes:

[0063] (1) Canal monitors the incremental log of the source database Binlog and generates a JSON format data push message queue;

[0064] (2) Incremental synchronization system consumes message records in the message queue;

[0065] (3) Filtering data according to message filtering rules;

[0066] (4) Perform distribution processing according to the preset distribution rules and send it to the corresponding sub-queue for consumption processing. Configure appropriate distribution rules for different businesses and different data volumes to achieve real-time data synchronization; among them, the preset distribution rules include the relationship rules between the front and back tables, the association rules of table field values, and the concurrent consumption rules of a single table;

[0067] (5) If an exception occurs, the retry mechanism is enabled, and the relevant exception data is added to the retry queue for retry operation.

[0068] See also Figure 2 and Figure 3 The present invention also provides an embodiment of a real-time synchronization method for massive data, which is applied to a real-time synchronization system for massive data. The method comprises:

[0069] Use Canal to obtain the incremental log of the source database, generate incremental data according to the incremental log, and push the incremental data as a message record to the message queue;

[0070] Using the incremental synchronization system to obtain the message records in the message queue, and distributing the message records to the corresponding message subqueues for concurrent consumption according to the preset distribution rules;

[0071] The message records in the message subqueue are processed by using an incremental synchronization system and then stored in a target database.

[0072] In step S100, the incremental data subscription and consumption service Canal is used to monitor the Binlog log generated by the source database and obtain the Binlog incremental log of the source database. This embodiment obtains the operation record in the Binlog incremental log of the source database through Canal, converts the operation record value into a string in JSON format to generate incremental data, and pushes the incremental data as a message record to the message queue.

[0073] This example uses Canal to obtain the incremental Binlog logs generated by the source database and pushes the data changes to the message queue (supports RocketMQ, Kafka, and RabbitMQ). To ensure the order of reading Binlog and writing MQ messages, you can configure a single Topic and single Queue mode.

[0074] It should be noted that the single Topic and single Queue mode is the default configuration of the Canal framework, which can be configured through the canal.properties file of Canal. The specific configuration is as follows:

[0075] #Configuration mode tcp, kafka, rocketMQ, rabbitMQ

[0076] canal.serverMode=rocketMQ;

[0077] #Configure the mq group address, group, topic and tag

[0078] rocketmq.producer.group=CANCAL;

[0079] rocketmq.customized.trace.topic=smarthome_topic;

[0080] rocketmq.namespace=***;

[0081] rocketmq.namesrv.addr=http: / / xxxxxx;

[0082] rocketmq.tag = data_tag;

[0083] In step S200, the message records in the message queue are obtained by using the incremental synchronization system, and the message records are distributed to corresponding message sub-queues for concurrent consumption according to preset distribution rules.

[0084] In this embodiment, the incremental synchronization system monitors message records and performs concurrent consumption. The preset distribution rules configured in the incremental synchronization system may include:

[0085] Pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules; among them, the pre- and post-table association rules distribute message records according to the current incrementally synchronized table, the table field value association rules distribute message records according to the fields in the table, and the single-table concurrent consumption rules distribute message records in parallel based on the modulus of the fields in a single table.

[0086] Specifically, the association rule between the front and back tables is to classify the tables according to the current incremental synchronization, put the basic table into the main queue for synchronization, and then classify it according to whether there is data in the front table in the business table. For example, if table A contains associated data in table B, then table B is the front table of table A, and the two tables A and B can be classified as one.

[0087] It should be noted that the main queue is the message queue for monitoring message records. The main queue can have multiple concurrent sub-queues, and then the sub-queues have sub-queues. The previous level sub-queue is the parent queue of the next level sub-queue.

[0088] The association rules of the preceding and following tables classify the tables that need to be synchronized in the current incremental data, classify and distribute them according to the association rules of the configuration table, put the basic table into the main queue for synchronization, and determine whether the basic table has a preceding table according to the business configuration table. If so, the basic table and all its preceding tables are classified as one category, that is, the basic table and the preceding table are added to the same message sub-queue.

[0089] See also Figure 4 For example, if table A contains associated data in table B, and table B contains associated data in table C, then table B is the pre-table of table A, table C is the pre-table of table B, and table C is also the pre-table of table A, that is, table A (basic table) has tables B and C as pre-tables, and these three tables are distributed to message queue 1 as the same category. In the pre- and post-table association rule, message records are distributed according to the relationship between the pre- and post-tables. If table A, table B, and table C are configured in message queue 1, the operation records of table A, table B, and table C are automatically forwarded to message queue 1 for processing.

[0090] Understandably, Figure 4 In the table, table B is the post-business association table of table C, and table A is the post-business association table of both table B and table C.

[0091] It should be noted that, according to the actual business configuration table, the association data between the basic table and the front table can have one or more levels.

[0092] Specifically, the table field value association rule is to classify according to certain fields in the table. For example, when a new user A is added, all subsequent business data related to the user are pushed to the same consumption queue to ensure the sequential consumption of all business data under the user, and there will be no concurrent consumption causing data confusion.

[0093] See also Figure 5 , the table field value association rule distributes the message record according to the table field value, obtains the table field value configured in the message record and the number of message queues for modulo operation, and determines the message queue to which the message record is to be forwarded. The specific rule can be: perform a modulo operation according to the configured classification field. For example, according to the user ID value classification, RocketMQ configures 8 Queue consumption queues. The modulo operation rule is to obtain the HASH value through the user ID and divide it by 8 to get the remainder, then you can get which specific queue the message record needs to be distributed to, and then implement the specified message to be sent to the specific specified Queue consumption queue according to RocketMQ's MessageQueueSelector interface.

[0094] Specifically, the single-table concurrent consumption rule distributes all records of the table to the concurrent consumption queue according to the configured table name, and distributes them to different queues through the specified field assignment values ​​to ensure the orderliness of the same record.

[0095] See also Figure 6 When the amount of data in a table for incremental synchronization (referred to as the synchronization table) is large or there is a backlog, the incremental synchronization system is used to configure the concurrent rules for a single table for concurrent consumption, and a non-empty field of the concurrent table is configured for modulo distribution to different message queues. While concurrent consumption, the order of the same record is guaranteed. The rule can be: If all the preceding business tables of the synchronization table are in the same parent queue, and the synchronization table has no subsequent business-related tables, the synchronization table can be configured to the concurrent consumption queue for concurrent consumption. For example, Figure 4 In the table A, the predecessor business tables of table A are table B and table C. When table B and table C are in the same parent queue, and table A is not the predecessor business table of other tables, that is, table A has no subsequent business association table, when the data volume of table A is large or there is a backlog, the modulus is taken according to a non-empty field of table A to distribute table A to the corresponding sub-queue for concurrent consumption.

[0096] In this embodiment, the three distribution rules of the pre- and post-table association rules, the table field value association rules and the single-table concurrent consumption rules can be used alone, or at least two of the pre- and post-table association rules, the table field value association rules and the single-table concurrent consumption rules can be used in combination according to the priority. The priority of the table field value association rules, the pre- and post-table association rules and the single-table concurrent consumption rules is from high to low. That is, the pre- and post-table association rules, the table field value association rules and the single-table concurrent consumption rules can be combined in any two or in full. For example, if table A and table B are related tables, the pre- and post-table association rules need to be used. If the amount of data in table B is large, the single-table concurrent consumption rules also need to be used. This situation corresponds to the simultaneous use of the pre- and post-table association rules and the single-table concurrent consumption rules. If the pre- and post-table association rules and the table field value association rules are used at the same time, the table field value association rules take precedence.

[0097] In the case where real-time synchronization of large amounts of data cannot be achieved based on the original canal middleware and MQ (Message Queue) method, this embodiment improves the overall synchronization program by setting pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules, combined with actual situation analysis, to ensure the real-time and effectiveness of synchronized data.

[0098] This embodiment combines the MQ characteristics to establish a main and sub-queue mechanism, performs relevant pre-processing on the data according to the main queue, and adapts the distribution rules of a message record when receiving it, matches the corresponding one or some distribution rules for the message record, and distributes and processes the message record according to the message distribution rules provided by the incremental synchronization system after parsing, and distributes the message record to the relevant sub-queues for processing, so as to achieve fast consumption and real-time synchronization. The incremental synchronization system provides 3 distribution rules, which can be used individually or in combination.

[0099] It should be noted that the characteristics of MQ are asynchronous, decoupled, and peak shaving. The main sub-queues are established mainly through the sequential messages, concurrent consumption, and message delay combination of the message queue.

[0100] In this embodiment, after using the incremental synchronization system to obtain the message record in the message queue, it also includes: if the message record is an update operation, then enter the filtering module for verification. Since the old platform (source database) may only update one or more unimportant fields for a certain record, the new platform (target database) does not use the field or useless values, the message filtering rules can be configured to filter the message records. When a table or certain fields corresponding to the message record are updated, the record is automatically ignored to avoid querying and operating the database, which can speed up the efficiency of data synchronization.

[0101] See also Figure 7 , the filtering module verification specifically includes:

[0102] S11: Determine whether a message filtering rule is configured. If so, execute S12. If not, execute the step of "distributing the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule";

[0103] S12: Verify whether the message record needs to be filtered. If so, end the operation. If not, execute the step of "distributing the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule".

[0104] It should be noted that, according to the actual synchronization situation, the insert and delete operations of the database must be executed and cannot be filtered; however, there may be fields that do not need to be paid attention to during the update operation. Therefore, when the message record is an update operation, certain message records can be filtered in a targeted manner.

[0105] This embodiment also includes: judging whether the message record is abnormal according to the preset abnormal conditions, and if abnormal, adding the abnormal message record to the retry queue and performing a retry operation. When an abnormal situation occurs such as a message record (data) cannot obtain an associated value, an abnormal trigger retry mechanism is formulated to judge whether the message record is abnormal according to the preset abnormal conditions, and if abnormal, send the abnormal data to the retry queue and perform a retry operation. When the number of retry operations reaches the configured number of retries (preset threshold), if there are still abnormal message records, the abnormal message path is subjected to an abnormal storage operation to analyze and process the abnormal data.

[0106] It should be noted that the retry queue is a concept in MQ, but the retry queue in this embodiment does not use the retry queue of MQ. Instead, a normal queue is defined as the retry queue, which is mainly used when an exception occurs in the synchronization of message records. The message record can be triggered according to the custom exception, and the message record can be placed in the retry queue for retry triggering.

[0107] It should be noted that MQ stands for Message Queue, and MQ messages are message records in the message queue. MQ is a message communication between applications. One end only publishes information to the queue, and the other end only reads messages from the queue. The publisher does not need to care about who reads the message, and the reader does not need to care about who publishes the message. Each does its own thing and does not interfere with each other. Common message queues include RabbitMQ, RocketMQ, and Kafka.

[0108] In this embodiment, before using the Canal cluster to obtain the incremental log of the source database, it also includes: deploying the Canal cluster, deploying the message queue, and configuring the table to be synchronized and the address of the pushed message queue through the Canal configuration file. Before using the incremental synchronization system to obtain the message record in the message queue, it also includes: deploying the incremental synchronization system and configuring the preset distribution rules.

[0109] In this embodiment, it is necessary to deploy message queues and Canal clusters, and configure the tables that need to be synchronized and the address of the message queues that need to be pushed through the Canal configuration file. Specifically, Canal configures canal.serverMode = RocketMQ / Kafka / RabbitMQ mode, and configures canal.mq.servers = MQ cluster address. Canal configures canal.mq.dynamicTopic = the table that needs to be synchronized, and configures canal.mq.partition = the partition value of the MySql record during the full export. Deploy the incremental synchronization system, configure the preset distribution rules, and configure the pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules according to the business configuration.

[0110] The real-time synchronization method for massive data provided in this embodiment uses Canal to monitor the source database and obtain incremental logs, generates incremental data based on the incremental logs and pushes them to the message queue, focusing on the data without paying attention to the upper-level business applications, avoiding association with the business, and having high decoupling; according to the configured preset distribution rules, the incremental synchronization system distributes message records to specific message sub-queues, flexibly configures data consumption, can speed up message consumption, and ensure the real-time synchronization of large amounts of data; through the retry mechanism and the exception reminder mechanism, the consistency of the incremental synchronization data is guaranteed.

[0111] See also Figure 8 Another embodiment of a real-time synchronization method for massive data provided by the present invention, the overall process is described in detail as follows:

[0112] (1) Deploy message queues and Canal clusters;

[0113] (2) Canal configures canal.serverMode = RocketMQ / Kafka / RabbitMQ mode, and configures canal.mq.servers = MQ cluster address;

[0114] (3) Canal configures canal.mq.dynamicTopic = the table to be synchronized, and configures canal.mq.partition = the partition value of the MySql record during the full export;

[0115] (4) Deploy the incremental synchronization system, configure the message record distribution rules, and configure the table association rules, field association rules, and single table concurrent consumption rules according to the business configuration;

[0116] (5) Start the program, and the main queue performs Canal monitoring and MQ consumption;

[0117] (6) If message filtering rules are configured, determine whether the message records need to be filtered. If necessary, filter the data according to the message filtering rules and distribute the message records to concurrent message subqueues for concurrent consumption;

[0118] (7) The incremental synchronization system performs relevant conversion and processing on the data (message records) in the message subqueue, converts the field values ​​of the source database into the corresponding field values ​​of the target database, and performs storage operations.

[0119] It should be noted that, generally speaking, when migrating data from the original platform to the new platform, the field values ​​of the source database and the target database may be different. Therefore, the field values ​​of the source database need to be converted into the field values ​​of the target database.

[0120] The present invention focuses on the transformation of queue distribution rules during MQ consumption. By consuming message records of message queues, the incremental synchronization system distributes message records to corresponding message sub-queues according to various distribution rules such as pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules, so as to achieve real-time synchronization of massive data.

[0121] The present invention configures preset distribution rules through the incremental synchronization system, distributes messages by the main queue, distributes to specific sub-queues according to the configured rules, flexibly configures data consumption, speeds up message consumption, and ensures the real-time nature of data synchronization. By monitoring the database through Canal, the focus is placed on the data, and there is no need to pay attention to upper-level business applications, avoiding association with the business, and having high decoupling. The consistency of incremental synchronization data is ensured through a retry mechanism and an abnormal reminder mechanism.

[0122] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0123] In the embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0124] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0125] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0126] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time synchronization system for massive data, characterized in that: include: Source database, Canal cluster, message queue cluster, incremental synchronization system, target database; Wherein, the source database generates incremental logs; The Canal cluster monitors the source database and obtains incremental logs, generates incremental data according to the incremental logs, and pushes the incremental data as message records to the message queue cluster; The message queue cluster receives the message record pushed by the Canal cluster; The incremental synchronization system monitors the message queue cluster and obtains message records, distributes the message records to corresponding message subqueues for concurrent consumption according to preset distribution rules, processes the message records in the message subqueues and stores them in the target database; The target database stores the message records processed by the incremental synchronization system; The preset distribution rules include: Pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules; Among them, the pre- and post-table association rule distributes message records according to the table of current incremental synchronization, the table field value association rule distributes message records according to the fields in the table, and the single-table concurrent consumption rule distributes message records in parallel according to the modulus of the fields in a single table.

2. A real-time synchronization method for massive data, characterized in that: Applied to a real-time synchronization system for massive data, the method includes: Use Canal to obtain the incremental log of the source database, generate incremental data according to the incremental log, and push the incremental data as a message record to the message queue; Using the incremental synchronization system to obtain the message records in the message queue, and distributing the message records to the corresponding message subqueues for concurrent consumption according to the preset distribution rules; Using the incremental synchronization system to process the message records in the message subqueue and then store them in the target database; The preset distribution rules include: Pre- and post-table association rules, table field value association rules, and single-table concurrent consumption rules; Among them, the pre- and post-table association rule distributes message records according to the table of current incremental synchronization, the table field value association rule distributes message records according to the fields in the table, and the single-table concurrent consumption rule distributes message records in parallel according to the modulus of the fields in a single table.

3. The real-time synchronization method of mass data according to claim 2, characterized in that: include: Using at least one of the preceding and following table association rules, the table field value association rules, and the single-table concurrent consumption rules according to the priority; Among them, the priorities of the table field value association rule, the pre- and post-table association rule, and the single-table concurrent consumption rule are from high to low.

4. The real-time synchronization method of mass data according to claim 2, characterized in that: After obtaining the message record in the message queue by using the incremental synchronization system, the method further includes: If the message record is an update operation, it will enter the filtering module for verification.

5. The real-time synchronization method of mass data according to claim 4, characterized in that: Filter module verification includes: S11: Determine whether a message filtering rule is configured. If so, execute S12. If not, execute step "distribute the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule"; S12: Verify whether the message record needs to be filtered. If so, end the operation. If not, execute the step of "distributing the message record to the corresponding message subqueue for concurrent consumption according to the preset distribution rule".

6. The real-time synchronization method of mass data according to claim 2, characterized in that: Also includes: Determine whether the message record is abnormal based on the preset abnormal conditions. If abnormal, add the abnormal message record to the retry queue and perform a retry operation.

7. The real-time synchronization method of mass data according to claim 6, characterized in that: After adding the abnormal message record to the retry queue and retrying, the following is also included: When the number of retry operations reaches the preset threshold, the abnormal message record is processed for abnormal storage operation.

8. The real-time synchronization method of mass data according to claim 2, characterized in that: Before using the Canal cluster to obtain the incremental logs of the source database, the following steps are also required: Deploy the Canal cluster and the message queue, and configure the tables to be synchronized and the address of the message queue to be pushed through the Canal configuration file.

9. The real-time synchronization method of mass data according to claim 2, characterized in that: Before using the incremental synchronization system to obtain the message record in the message queue, the method further includes: Deploy the incremental synchronization system and configure preset distribution rules.

Images