Data synchronization method, device, equipment and computer storage medium

Abstract

The present invention relates to the technical field of financial technology (Fintech), and discloses a data synchronization method, which includes: obtaining multiple WAL task lists in HBase, and assigning each of the WAL task lists to a DRC server corresponding to each of the task lists; determining a target DRC server corresponding to a target WAL task list in each of the WAL task lists, obtaining a target log file in the target WAL task list according to the target DRC server, and copying the target log file content of the target log file to Kafka; if it is detected that the log file in Kafka is consumed, the consumption data of the consumption is written into the standby cluster of HBase. The present invention also discloses a data synchronization device, equipment and a computer storage medium. The present invention realizes the separation of replication from HBase cluster and improves the stability of HBase cluster.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and in particular to a data synchronization method, device, equipment and computer storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are applied in the financial field. The traditional financial industry is gradually transforming to Fintech. However, due to the security and real-time requirements of the financial industry, higher requirements are also placed on data synchronization technology. At present, HBase (open source database) uses the typical LSM (Log-Structured Merge-Tree) data structure when writing data. After the server receives the write request, it will first write it to Memstore (memory) and write it sequentially to the WAL log (HDFS file). The WAL log ensures that the data is actually written to the disk, even if the RS service is down. In the event of a node failure, the WAL log can be used to restore the memory data that has not been persisted. HBASE itself provides a replication function, HBASE Replication, but all HBASE Replication work is completed in the HBASE host group, which will affect the HBASE cluster. For example, when HBASE has a large amount of data to write, it will compete with the HBASE Replication thread for CPU (central processing unit), memory and disk resources, resulting in increased replication time. Replication will also cause data read and write performance to decline, making HBase less stable during data synchronization. Therefore, how to improve the stability of HBase during data synchronization has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the present invention is to propose a data synchronization method, device, equipment and computer storage medium, aiming to solve the technical problem of how to improve the stability of hbase in data synchronization.

[0004] To achieve the above object, the present invention provides a data synchronization method, which comprises the following steps:

[0005] Get multiple wal task lists in hbase, and assign each wal task list to the drc server corresponding to each task list;

[0006] Determine the target drc server corresponding to the target wal task list in each of the wal task lists, obtain the target log file in the target wal task list according to the target drc server, and copy the target log file content of the target log file to kafka;

[0007] If it is detected that the log file in Kafka is consumed, the consumption data is written into the standby cluster of HBase.

[0008] Optionally, the step of copying the target log file content of the target log file to Kafka includes:

[0009] If there are multiple target log files, traverse each target log file in turn, determine the log sequence number of the current target log file currently traversed in the target wal task list, and obtain the historical sequence number of the region corresponding to the current target log file that has been written into Kafka, and detect whether the historical sequence number is greater than or equal to the log sequence number;

[0010] If so, the target log file content of the current target log file is copied to the kafka until the traversal of each target log file is completed.

[0011] Optionally, after the step of detecting whether the historical sequence number is greater than or equal to the log sequence number, the following steps are included:

[0012] If not, stop copying the current target log file to Kafka and output a prompt message indicating that the copying failed until the traversal of each target log file is completed.

[0013] Optionally, the step of obtaining the historical sequence number of the region corresponding to the current target log file that has been written into Kafka includes:

[0014] Based on the region corresponding to the current target log file, determine the historical log file that was last received in the kafka from the current moment, and obtain the historical file position and historical log sequence number of the historical log file in the target wal task list, and use the historical log sequence number and the historical file position as the historical serial number corresponding to the kafka.

[0015] Optionally, the step of obtaining the target log file in the target WAL task list according to the target DRC server includes:

[0016] All log files in the target wal task list are read according to the target drc server, and each log file is filtered to obtain the target log file.

[0017] Optionally, the steps of obtaining multiple WAL task lists in HBase include:

[0018] Based on a preset time period, multiple wal file lists in hbase are read, invalid files in each wal file list are deleted, and the wal file lists after deletion are used as wal task lists in hbase.

[0019] Optionally, after the step of obtaining multiple WAL task lists in HBase and allocating each WAL task list to a DRC server corresponding to each task list, the following steps are included:

[0020] Register each of the wal task lists to the configuration management and monitoring server corresponding to the hbase, and monitor each of the wal task lists through the configuration management and monitoring server;

[0021] If there is an abnormal wal task list in each of the wal task lists, an alarm is performed.

[0022] In addition, to achieve the above object, the present invention also provides a data synchronization device, the data synchronization device comprising:

[0023] An acquisition unit is used to acquire multiple wal task lists in hbase and distribute each wal task list to a drc server corresponding to each task list;

[0024] A copying unit is used to determine a target DRC server corresponding to a target WAL task list in each of the WAL task lists, obtain a target log file in the target WAL task list according to the target DRC server, and copy the target log file content of the target log file to Kafka;

[0025] The writing unit is used to write the consumption data of the consumption into the standby cluster of HBase if it is detected that the log file in Kafka is consumed.

[0026] In addition, to achieve the above-mentioned purpose, the present invention also provides a data synchronization device, which includes: a memory, a processor, and a data synchronization program stored on the memory and executable on the processor, and the data synchronization program implements the steps of the data synchronization method described above when executed by the processor.

[0027] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a data synchronization program is stored, and when the data synchronization program is executed by a processor, the steps of the data synchronization method described above are implemented.

[0028] The present invention obtains multiple WAL task lists in HBase, and distributes each WAL task list to a DRC server corresponding to each task list; determines the target DRC server corresponding to the target WAL task list in each WAL task list, obtains the target log file in the target WAL task list according to the target DRC server, and copies the target log file content of the target log file to Kafka; if it is detected that the log file in Kafka is consumed, the consumption data of the consumption is written into the standby cluster of HBase. By distributing each task list in HBase to the corresponding DRC servers, and copying the target log file to Kafka through the target DRC server, and writing the consumption data into the standby cluster of HBase when the log file in Kafka is consumed, the prior art avoids the phenomenon that the overall performance of HBase is unstable when data reading and writing and HBase copy operations exist simultaneously in HBase, and the stability of HBase during data synchronization is improved by separating the copy operation of HBase from HBase itself. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the structure of a data synchronization device in a hardware operating environment involved in an embodiment of the present invention;

[0030] Figure 2 This is a flow chart of a first embodiment of a data synchronization method of the present invention;

[0031] Figure 3 A schematic diagram of a device module of a data synchronization device of the present invention;

[0032] Figure 4 It is a schematic diagram of the process of consumer data consumption in the data synchronization method of the present invention;

[0033] Figure 5 This is a schematic diagram of the architecture of hbase in the data synchronization method of the present invention;

[0034] Figure 6 It is a schematic diagram of the process flow running in the data synchronization method of the present invention.

[0035] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0037] like Figure 1 As shown, Figure 1It is a schematic diagram of the structure of a data synchronization device in a hardware operating environment involved in an embodiment of the present invention.

[0038] The data synchronization device in the embodiment of the present invention may be a PC or a server device, on which a Java virtual machine runs.

[0039] like Figure 1 As shown, the data synchronization device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0040] Those skilled in the art will understand that Figure 1 The data synchronization device structure shown in the figure does not constitute a limitation on the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0041] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a data synchronization program.

[0042] exist Figure 1 In the data synchronization device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the data synchronization program stored in the memory 1005 and perform the operations in the following data synchronization method.

[0043] Based on the above hardware structure, an embodiment of the data synchronization method of the present invention is proposed.

[0044] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a data synchronization method of the present invention, the method comprising:

[0045] Step S10, obtaining multiple wal task lists in hbase, and assigning each wal task list to a drc server corresponding to each task list;

[0046] In this embodiment, HBASE is a distributed KV (key-value, data structure) storage engine for storing massive data. WAL (Write-Ahead Logging, pre-write log system) pre-write log is WAL log, when data is written to HBASE cluster, the data will be written sequentially into memory and log file, the log file is called WAL log, WAL log ensures that data is actually written to disk, so as to restore the memory data that is not persisted when the node fails, and WAL log is stored in HDFS. WalGroup is WAL group, RegionServer (hbase service) will group the logs under the current server according to the configuration and write them into different wal log files. DRC (Data Replication Center, data replication center) provides services such as real-time data synchronization of heterogeneous databases and data record change subscription. Kafka is a high-throughput distributed message publishing and subscription system. Region is the basic unit of HBase data management. Table maintenance, data storage, cluster load balancing, etc. are all based on region. HBase cluster service, RegionServer (RS) is deployed on a physical machine. HDFS is a distributed file system, and the data of HBASE cluster is stored in the hdfs system.

[0047] In this embodiment, the replication of HBase cluster data is mainly separated from the HBase cluster itself through HBase DRC, which improves the overall performance, stability and reliability of HBase. HBase DRC mainly includes DRC service, configuration management and monitoring, message middleware Kafka, and various business consumers. And the DRC service is a distributed service, which is operated by multiple machines at the same time, and a server is selected as the leader to scan the WAL log and assign the WAL task list. And the DRC server cluster selects a DRC server as the leader server through Zookeeper (a reliable coordination system for distributed systems), and other DRC servers are task servers. Among them, the leader server is responsible for task discovery and task assignment. The Task server is responsible for reading and writing log files to Kafka. The Task server will be registered in the TaskServer service list in Zookeeper. The taskserver will record the assigned task list. For example, the content stored in the "172.10.10.10" node is "node3%2C16020%2C157695485123853.1563587645638", that is, the "172.10.10.10" node processes the "node3%2C16020%2C157695485123853.1563587645638" wal log file.

[0048] In this embodiment, the wal log in hdfs can be scanned regularly to determine whether there is a new wal log file, and the two directories / hbase / WALs and / hbase / oldWAL need to be scanned in hdfs, and the scanned files (such as .meta. and split files, where split is a temporary file generated when region is split) are filtered, only the wal log file is retained, and the filtered wal log file is registered to the configuration management and monitoring service to obtain multiple wal task lists, and the leader server in the DRC server distributes each wal task list to each drc server, that is, the task server, according to the task allocation thread.

[0049] Among them, after the leader server reads the wal log file in hdfs, it can record it to the "storage.drc-hbase.tasks" node of zookeeper. The data content of this node is the processing progress information of the current task. For example, the data content of the wal log file "node2%2C16020%2C157695485963854.regiongroup-0.156352874532" stored in zookeeper is:

[0050]

[0051]

[0052] Among them, walname is the name of the wal log file. Status is an optional task status, including initialization, assigned, tentative execution, execution completed, execution failed, etc. Taskserver is the machine that processes the current task. Position is the location where the current wal file is read, which is convenient for breakpoint continuation. In addition, the data that the task server needs to execute needs to be written in "storage.drc-hbase.taskServer.taskIP". The data format can be as follows:

[0053]

[0054] Step S20, determining the target drc server corresponding to the target wal task list in each of the wal task lists, obtaining the target log file in the target wal task list according to the target drc server, and copying the target log file content of the target log file to kafka;

[0055] After each wal task list is assigned to the drc server, the non-leader nodes in the DRC cluster will process their own tasks. That is, each drc server (that is, task server) that receives the wal task list processes the task separately without affecting each other. Therefore, the target wal task list in each wal task list and the target drc server corresponding to the target wal task list can be determined first, and then the target log file content of the target log file in the target wal task list can be copied to kafka according to the target drc server, that is, the target log file will be read, and the read target log file content will be written to kafka to complete the copy process, wherein the target log file content can be a modification record of data line by line.

[0056] That is, taskserver will monitor whether there is any data change in the "storage.drc-hbase.taskServer.taskIP" node of zookeeper. If there is any data change, it is determined that the data of the node is the task that the current task server needs to process. At this time, the target drc server will read the data of the node, obtain the target wal task list, and check the target wal task list, that is, read the content of "storage.drc-hbase.tasks.wal file name", check whether the current wal log task is assigned to this taskServer, read the execution progress of the current WAL task: position, check the file integrity, etc. After the check of the target wal task list is completed, the log file in the target wal task list will be read from position. If the position of the log file does not exist, it is determined that the log file is a new task, and the default position = 0. Among them, the log format of the read log file can be as follows:

[0057]

[0058]

[0059] Therefore, after reading the log file, it is necessary to filter the log file, such as removing the logs where "table.nameWithNamespaceInclAsString" is "hbaes:meta" and some "EVENT" type logs, such as split, compact and other event logs. Only valid data logs, that is, the target log file, are retained, and then the target log file is written to kafka. At the same time, the position and sequenceid of the target log file will be reported to the zookeeper of the configuration management center, recorded as the sequence number lastPushedSequenceId. Among them, position is the position of the current data in the wal file. When modifying the data of the "storage.drc-hbase.tasks.wal file name" node, the position will be updated. Sequenceid is the log sequence number at the region level, and the logs generated by each region are incremental, even if migration, splitting, merging and other operations occur. The sequenceids of different regions do not affect each other. Modify the data of "storage.drc-hbase.sequence.table name.region name" and write lastPushedSequenceId, where lastPushedSequenceId is the sequenceId in the wal log successfully written to Kafka.

[0060] It should also be noted that, in this embodiment, when the log is copied, it is executed serially at the region level, and the sequence number lastPushedSequenceId of the last record sent to Kafka in the current region will be queried in Zookeeper; the lastPushedSequenceId will be compared with the sequenceId currently read from the WAL. If the lastPushedSequenceId is greater than or equal to the sequenceId, the copy is allowed, otherwise the current copy thread is blocked, and a periodic detection is performed to determine whether the copy can continue.

[0061] Step S30: If it is detected that the log file in Kafka is consumed, the consumption data is written into the standby cluster of HBase.

[0062] In this embodiment, consumers include consumers that write to the slave hbase cluster, and also include different business parties that subscribe to kafka data. When the slave HBASE cluster consumer consumes data in kafka, the consumer consumes the data of the specified table or the full amount of data on demand according to its own business needs. After reading the data, the single-threaded consumption data is written to the slave cluster. That is, when it is detected that the log file in kafka is consumed, the consumed consumption data is written to the backup cluster of hbase or other heterogeneous storage media. Among them, when multi-threaded consumption is performed on the consumer side, the data read from kafka will be routed to different queues according to "tablename+rowkey", and then each queue will still be written to kafka in single-threaded order, so as to ensure the orderliness of the row level, thereby ensuring the data consistency of the overall DRC data synchronization.

[0063] In addition, when writing consumption data to the standby cluster of HBase, it is necessary to perform secondary conversion according to the operation type of the data in Kafka. For example, when the operation type is put, the consumption data is encapsulated as data in the format of "put'table name','row key','column family name:column name','column value'" to perform the write operation. And when writing consumption data to elasticsearch or hive, you can Figure 4 As shown in the figure: After copying the target log file in the drc service to the kafka queue, each consumer can store the consumption data generated by the consumption in the kafka queue in different storage media. For example, consumer sink1 stores its own consumption data in hbase slave data; consumer sink2 stores its own consumption data in hive; and consumer sink3 stores its own consumption data in es (Elasticsearch, search server). Among them, data consumption in kafka is used to write to the standby cluster or other heterogeneous storage media. Consumers continuously consume kafka data and write it to the standby cluster.

[0064] In addition, to assist in understanding the principle of data import in this embodiment, an example is given below.

[0065] like Figure 5As shown, it is the overall architecture diagram of data import in this embodiment, including Hbase cluster, DRC service, configuration management and monitoring and message middleware kafka. In addition, the hbase cluster contains multiple RS and HDFS, and there are multiple wal1, wal2, wal3 and wal4 in HDFS. The DRC service includes task discovery and registration module, task allocation module, task initialization module, wal data reading module, data processing and filtering module and data output module, while configuration management and monitoring include hbase cluster monitoring, wal log file change monitoring and reporting, wal replication progress monitoring and reporting and replication delay monitoring and reporting. Its operation process is as follows Figure 6 As shown in the figure, when data import starts, task discovery and initialization operations are performed first, then task execution is performed, and finally data consumption is performed, that is, the list of wal log files on hdfs is read in hbase first, and then invalid files in these wal log file lists are removed, that is, the logs being split are filtered out, and the remaining WAL logs are registered to the configuration management and monitoring service. The configuration management and monitoring service uses Zookeeper to maintain task information, and the wal logs registered to the configuration service are assigned to different drc servers for processing through the task allocation thread in the drc server, and each drc server does not affect each other when processing its own wal log. When the drc server processes the wal log, it first uses walreader to read the wal log, and after reading the wal log, it performs log filtering, and then writes the filtered log to kafka, that is, performs a copy operation, and records the copy progress and offset. At the same time, consumers can consume the wal log data in kafka and write it in batches to the hbase standby cluster until the end.

[0066] In this embodiment, multiple wal task lists in hbase are obtained, and each wal task list is assigned to the drc server corresponding to each task list; the target drc server corresponding to the target wal task list in each wal task list is determined, the target log file in the target wal task list is obtained according to the target drc server, and the target log file content of the target log file is copied to kafka; if it is detected that the log file in kafka is consumed, the consumption data of the consumption is written to the standby cluster of hbase. By assigning each task list in hbase to the corresponding drc server, and copying the target log file to kafka through the target drc server, and writing the consumption data to the standby cluster of hbase when the log file in kafka is consumed, the prior art avoids the phenomenon that the overall performance of hbase is unstable when data reading and writing and hbase copy operations exist simultaneously in hbase, and the stability of hbase during data synchronization is improved by separating the copy operation of hbase from hbase itself.

[0067] Furthermore, based on the first embodiment of the data synchronization method of the present invention, a second embodiment of the data synchronization method of the present invention is proposed. This embodiment is a refinement of step S20 of the first embodiment of the present invention, the step of copying the target log file content of the target log file to Kafka, including:

[0068] Step a: If there are multiple target log files, traverse each target log file in turn, determine the log sequence number of the current target log file in the target wal task list, obtain the historical sequence number of the region corresponding to the current target log file that has been written to Kafka, and detect whether the historical sequence number is greater than or equal to the log sequence number;

[0069] When it is found that there are multiple target log files in the target wal task list, each target log file can be traversed in turn, and the log sequence number sequenceid of the current target log file in the target wal task list can be determined, and then all the sequence numbers sent to kafka by the current region to which the current target log file is applied can be obtained from zookeeper, and the sequence number received most recently from these sequence numbers can be determined, and used as the historical sequence number, and then the historical sequence number can be detected whether it is greater than or equal to the log sequence number, and different operations can be performed according to different detection results. Among them, the sequence number includes the position and sequenceid of the log file corresponding to the sequence number in the wal task list.

[0070] Step b: If yes, the log file content of the current target log file is copied to the kafka until the traversal of each target log file is completed.

[0071] When it is found that the historical sequence number is greater than or equal to the log sequence number, the log file content of the current log file can be directly copied to Kafka through the current region, and the other target log files in the target WAL task list can be detected in the same way until the traversal of each target log file is completed.

[0072] In this embodiment, when there are multiple target log files and the historical sequence number of Kafka is greater than or equal to the log sequence number of the current target log file, the current target log file can be copied to Kafka, thereby ensuring the normal copy operation in HBase.

[0073] Further, after the step of detecting whether the historical sequence number is greater than or equal to the log sequence number, the following steps are included:

[0074] Step c: if not, stop copying the current target log file to Kafka and output a prompt message of copy failure until the traversal of each target log file is completed.

[0075] If the historical sequence number is less than the log sequence number, the current copying thread is blocked. At this time, the current file copying to Kafka will be stopped, and a copy failure prompt will be output to inform the user. At the same time, the thread will be detected regularly to determine whether the copy operation can be performed. If it can, the current target log file will continue to be copied until all log files are traversed.

[0076] In this embodiment, when it is determined that the historical sequence number is less than the log sequence number, the current target log file is stopped from being copied to Kafka, and a prompt message of copy failure is output, thereby avoiding the phenomenon of losing the overall performance of HBase due to thread blocking.

[0077] Furthermore, the step of obtaining the historical sequence number of the region corresponding to the current target log file that has been written into Kafka includes:

[0078] Step d, based on the region corresponding to the current target log file, determine the historical log file that was last received in the kafka from the current moment, and obtain the historical file position and historical log sequence number of the historical log file in the target wal task list, and use the historical log sequence number and the historical file position as the historical serial number corresponding to the kafka.

[0079] In this embodiment, when obtaining the historical sequence number corresponding to Kafka, it is necessary to first determine the log file that was received last time in Kafka, that is, the historical log file, according to the region corresponding to the current target log file, that is, to query the sequence number and log file of the last message sent to Kafka in the current region in Zookeeper. Then determine the historical file position and historical log sequence number of the historical log file in the target WAL task list, and directly use this historical log sequence number and historical file position as the historical sequence number of Kafka.

[0080] In this embodiment, by determining the historical file position and historical log sequence number of the historical log file in Kafka in the target WAL task list and using them as the historical sequence number, the accuracy of the acquired historical sequence number is guaranteed.

[0081] Further, the step of obtaining the target log file in the target wal task list according to the target drc server includes:

[0082] Step e: read all log files in the target wal task list according to the target drc server, and filter each log file to obtain the target log file.

[0083] In this embodiment, when obtaining the target log file according to the target drc server, it is necessary to first read all the log files in the target wal task list according to the target drc server, and check the content of each log file in turn, such as checking the integrity of the content of the log file. And at the same time of detection, each log file will be filtered, such as removing the logs of "table.nameWithNamespaceInclAsString" being "hbaes:meta" and some "EVENT" type, such as split, compact and other event logs. And the log file after filtering is used as the target log file.

[0084] In this embodiment, all log files in the target WAL task list are read according to the target DRC server, and each log file is filtered to obtain the target log file, thereby ensuring the accuracy of the obtained target log file.

[0085] Furthermore, the steps of obtaining multiple WAL task lists in HBase include:

[0086] Step f, reading multiple wal file lists in hbase based on a preset time period, deleting invalid files in each of the wal file lists, and using the wal file lists that have been deleted as the wal task list in hbase.

[0087] In this embodiment, multiple wal task lists in hbase can be obtained periodically, that is, multiple wal file lists in hbase can be read according to the time period set in advance by the user, and after reading each wal file list, the invalid files read will be deleted, such as filtering out .meta. files, etc., and the wal file list that has been deleted will be used as the wal task list in hbase.

[0088] In this embodiment, a plurality of wal file lists are obtained based on a preset time period, and invalid files are deleted to obtain a wal task list, thereby ensuring the accuracy of the obtained wal task list.

[0089] Further, after the step of obtaining multiple wal task lists in hbase and allocating each wal task list to a drc server corresponding to each task list, the method further includes:

[0090] Step g, registering each of the wal task lists to the configuration management and monitoring server corresponding to the hbase, and monitoring each of the wal task lists through the configuration management and monitoring server;

[0091] In this embodiment, after determining each wal task list, it is necessary to register the logs in each wal task list to the configuration management and monitoring server corresponding to hbase, and monitor each wal task list through the configuration management and monitoring server. It should be noted that the configuration management and monitoring can monitor the hbase cluster, monitor and report wal log file changes, monitor and report wal replication progress, monitor and report replication delay, etc.

[0092] Step h: if there is an abnormal wal task list in each of the wal task lists, an alarm is processed.

[0093] When it is found through judgment that there are wal task lists with abnormalities (such as traces of hacker intrusion, etc.) in each wal task list, that is, abnormal wal task list, an alarm will be actively processed. In this embodiment, drc service failures can also be detected, that is, different drc servers will seize the distributed mutex lock through zookeeper, and the DRC service that seizes the lock is the leader, and the others are flowers. The leader not only scans the WAL log to assign tasks, but is also responsible for monitoring the survival of the entire DRC service cluster to detect drc service failures.

[0094] In this embodiment, each wal task list is registered with the configuration management and monitoring server, and an alarm is processed when an abnormal wal task list is monitored, thereby ensuring the normal operation of hbase.

[0095] In addition, it should be noted that, in this embodiment, after the doc server reads the wal log content, the operation type of the wal log content will be recorded together, where the operation type can be as follows:

[0096]

[0097] When the doc server reads the wal log content with the operation type, it will use the combination of "tablename+rowkey" to calculate the hash value and route it to different kafka partitions to achieve table-level data independence and row-level data order, ensuring data reliability and consistency.

[0098] The present invention also provides a data synchronization device, referring to Figure 3 , the data synchronization device comprises:

[0099] The acquisition unit A10 is used to acquire multiple wal task lists in the hbase, and distribute each of the wal task lists to the drc server corresponding to each of the task lists;

[0100] A copying unit A20 is used to determine a target drc server corresponding to a target wal task list in each of the wal task lists, obtain a target log file in the target wal task list according to the target drc server, and copy the target log file content of the target log file to kafka;

[0101] The writing unit A30 is used to write the consumption data into the standby cluster of HBase if it is detected that the log file in Kafka is consumed.

[0102] Optionally, the replication unit A20 is further used for:

[0103] If there are multiple target log files, traverse each target log file in turn, determine the log sequence number of the current target log file currently traversed in the target wal task list, and obtain the historical sequence number of the region corresponding to the current target log file that has been written into Kafka, and detect whether the historical sequence number is greater than or equal to the log sequence number;

[0104] If so, the target log file content of the current target log file is copied to the kafka until the traversal of each target log file is completed.

[0105] Optionally, the replication unit A20 is further used for:

[0106] If not, stop copying the current target log file to Kafka and output a prompt message indicating that the copying failed until the traversal of each target log file is completed.

[0107] Optionally, the replication unit A20 is further used for:

[0108] Based on the region corresponding to the current target log file, determine the historical log file that was last received in the kafka from the current moment, and obtain the historical file position and historical log sequence number of the historical log file in the target wal task list, and use the historical log sequence number and the historical file position as the historical serial number corresponding to the kafka.

[0109] Optionally, the replication unit A20 is further used for:

[0110] All log files in the target wal task list are read according to the target drc server, and each log file is filtered to obtain the target log file.

[0111] Optionally, the acquisition unit A10 is further used for:

[0112] Based on a preset time period, multiple wal file lists in hbase are read, invalid files in each wal file list are deleted, and the wal file lists after deletion are used as wal task lists in hbase.

[0113] Optionally, the acquisition unit A10 is further used for:

[0114] Register each of the wal task lists to the configuration management and monitoring server corresponding to the hbase, and monitor each of the wal task lists through the configuration management and monitoring server;

[0115] If there is an abnormal wal task list in each of the wal task lists, an alarm is performed.

[0116] The methods executed by the above-mentioned program units can refer to the various embodiments of the data synchronization method of the present invention, and will not be described in detail here.

[0117] The invention also provides a computer storage medium.

[0118] The computer storage medium of the present invention stores a data synchronization program, and when the data synchronization program is executed by a processor, the steps of the data synchronization method described above are implemented.

[0119] Among them, the method implemented when the data synchronization program running on the processor is executed can refer to the various embodiments of the data synchronization method of the present invention, and will not be repeated here.

[0120] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0121] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0123] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that: The data synchronization method comprises the following steps: Read multiple wal file lists in hbase based on a preset time period, delete invalid files in each wal file list, use the wal file lists that have been deleted as wal task lists in hbase, and assign each wal task list to a drc server corresponding to each task list; The drc server provides real-time data synchronization of heterogeneous databases and data record change subscription services; Determine the target drc server corresponding to the target wal task list in each of the wal task lists, obtain the target log file in the target wal task list according to the target drc server, and copy the target log file content of the target log file to kafka; If it is detected that the log file in Kafka is consumed, the consumption data is written into the standby cluster of HBase.

2. The data synchronization method according to claim 1, characterized in that: The step of copying the target log file content of the target log file to Kafka comprises: If there are multiple target log files, traverse each target log file in turn, determine the log sequence number of the current target log file currently traversed in the target wal task list, and obtain the historical sequence number of the region corresponding to the current target log file that has been written into Kafka, and detect whether the historical sequence number is greater than or equal to the log sequence number; If so, the target log file content of the current target log file is copied to the kafka until the traversal of each target log file is completed.

3. The data synchronization method according to claim 2, characterized in that: After the step of detecting whether the historical sequence number is greater than or equal to the log sequence number, the method further comprises: If not, stop copying the current target log file to Kafka and output a prompt message indicating that the copying failed until the traversal of each target log file is completed.

4. The data synchronization method according to claim 2, characterized in that: The step of obtaining the historical sequence number of the region corresponding to the current target log file that has been written into Kafka includes: Based on the region corresponding to the current target log file, determine the historical log file that was last received in the kafka from the current moment, and obtain the historical file position and historical log sequence number of the historical log file in the target wal task list, and use the historical log sequence number and the historical file position as the historical serial number corresponding to the kafka.

5. The data synchronization method according to claim 1, characterized in that: The step of obtaining the target log file in the target WAL task list according to the target DRC server includes: All log files in the target wal task list are read according to the target drc server, and each log file is filtered to obtain the target log file.

6. The data synchronization method according to any one of claims 1 to 5, characterized in that: After the step of allocating each of the WAL task lists to the DRC servers corresponding to each of the task lists, the method further comprises: Register each of the wal task lists to the configuration management and monitoring server corresponding to the hbase, and monitor each of the wal task lists through the configuration management and monitoring server; If there is an abnormal wal task list in each of the wal task lists, an alarm is performed.

7. A data synchronization device, characterized in that: The data synchronization device comprises: The acquisition unit is used to read multiple wal file lists in hbase based on a preset time period, delete invalid files in each wal file list, use the wal file list after deletion as the wal task list in hbase, and distribute each wal task list to the drc server corresponding to each task list; the drc server provides real-time data synchronization of heterogeneous databases and data record change subscription services; A copying unit is used to determine a target DRC server corresponding to a target WAL task list in each of the WAL task lists, obtain a target log file in the target WAL task list according to the target DRC server, and copy the target log file content of the target log file to Kafka; The writing unit is used to write the consumption data of the consumption into the standby cluster of HBase if it is detected that the log file in Kafka is consumed.

8. A data synchronization device, characterized in that: The data synchronization device comprises: a memory, a processor, and a data synchronization program stored in the memory and executable on the processor, wherein the data synchronization program implements the steps of the data synchronization method according to any one of claims 1 to 6 when executed by the processor.

9. A computer storage medium, characterized in that The computer storage medium stores a data synchronization program, and when the data synchronization program is executed by the processor, the steps of the data synchronization method according to any one of claims 1 to 6 are implemented.

Images