Heterogeneous data synchronization method, equipment and medium

Abstract

The invention discloses a heterogeneous data synchronization method and device and a medium, and relates to the technical field of database synchronization, the method comprises the following steps: determining a source database corresponding to heterogeneous data, and collecting a real-time load index of the source database; the number of the source databases is one or more; calling a change capture engine corresponding to the source database, dynamically adjusting the operation mode of the change capture engine according to the real-time load index, and capturing a change event; and classifying the change events, determining corresponding message queues and sending the message queues so as to transmit the change events to a target end through the message queues. Through a multi-dimensional dynamic adaptation mechanism, the problems of performance stiffness and compatibility of a traditional data synchronization scheme in a heterogeneous environment are systematically solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of database synchronization, in particular to a heterogeneous data synchronization method, device and medium. BACKGROUND

[0002] With the diversified development of database technology, many enterprises begin to comprehensively use multiple types of database systems in their business data management to meet different business needs, including traditional relational databases such as Oracle and MySQL, non-relational databases (NoSQL), and new analytical databases. Among them, NoSQL databases are mainly used for high concurrency and massive data problems, and at the same time, new analytical databases are introduced to optimize online analytical processing (OLAP) scenarios. In the case of mixed use of multiple database systems, heterogeneous data is easily generated, and due to the scene needs of different business lines of enterprises, data needs to be synchronized across databases, so the heterogeneous data needs to be synchronized.

[0003] However, the MySQL native replication technology is limited to data synchronization between MySQL clusters, and has limited processing capacity for heterogeneous database type conversion. In the traditional field of heterogeneous data synchronization, there are two main methods. The first method is to use ETL tools for periodic scanning synchronization, but this method has a synchronization delay problem and cannot meet the real-time synchronization demand; the second method is to use a general change data capture (CDC) tool to realize data synchronization, but the deployment process of the CDC tool is relatively complex and requires high system resources, resulting in increased costs. SUMMARY

[0004] In order to solve the above problems, the present application provides a heterogeneous data synchronization method, comprising:

[0005] determining the source database corresponding to the heterogeneous data, collecting the real-time load index of the source database; the number of source databases is 1 or more;

[0006] calling the change capture engine corresponding to the source database, dynamically adjusting the running mode of the change capture engine according to the real-time load index, and capturing the change event;

[0007] classifying the change event, determining the corresponding message queue and sending, so as to transmit the change event to the target end through the message queue.

[0008] On the other hand, the present application also provides a heterogeneous data synchronization device, comprising:

[0009] at least one processor; and,

[0010] a memory in communication connection with the at least one processor; wherein,

[0011] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a heterogeneous data synchronization method as described in the above examples.

[0012] In another aspect, the application further provides a non-volatile computer storage medium storing computer executable instructions configured to perform a heterogeneous data synchronization method as described in the above examples.

[0013] The heterogeneous data synchronization method provided by the application can bring the following beneficial effects:

[0014] Through the multi-dimensional dynamic adaptation mechanism, the performance rigidity and compatibility problems of the traditional data synchronization scheme in the heterogeneous environment are systematically solved.

[0015] Firstly, full-scene coverage of different databases is realized, and through the special change capture engine and load-aware strategy, data integrity can be ensured when the load is light, and system overhead can be reduced through merging algorithm and key field filtering when the load is heavy, so that the resource utilization is significantly improved.

[0016] Secondly, through the semantic event classification and multi-queue transmission mechanism, the system can intelligently select the transmission channel according to the change type and data characteristics, ensure the strong consistency of the core data, allow non-critical changes to pass through the asynchronous queue to improve the overall throughput, and realize the optimal balance between data synchronization quality and system performance. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings described herein are used to provide further understanding of the application, form a part of the application, and the illustrative embodiments of the application and their descriptions are used to explain the application, and do not constitute improper limitations on the application. In the drawings:

[0018] Figure 1 Fig. 1 is a flowchart of a heterogeneous data synchronization method in an embodiment of the application;

[0019] Figure 2 Fig. 3 is a flowchart of a change capture engine for a MySQL database in an embodiment of the application;

[0020] Figure 3 Fig. 5 is a schematic diagram of a heterogeneous data synchronization device in an embodiment of the application. DETAILED DESCRIPTION

[0021] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0022] The technical solutions provided by the embodiments of the present application will be described in detail below in connection with the drawings.

[0023] As shown in the Figure 1 , the present application provides a heterogeneous data synchronization method, comprising:

[0024] S101: Determine the source database corresponding to the heterogeneous data, and collect the real-time load indicators of the source database; the number of the source database is one or more.

[0025] Specifically, the heterogeneous data and the source database of the heterogeneous data are obtained, the source database type is determined through the configuration file or the metadata registration center of the source database, the types of the source database include a log type database, a CDC interface type database, a cloud native type database and a non-log type database, and the number of the source database is one or more.

[0026] It should be noted that the log type database records all data changes through transaction logs (such as Binlog, Redo Log); the CDC interface type database collects the change data stream by providing a native CDC interface or a change stream service; the cloud native type database depends on the managed change stream service provided by the cloud vendor to obtain the change data stream; and the non-log type database lacks a native log or a CDC interface and needs to capture changes through external means.

[0027] Further, the load indicator data is collected in real time through the built-in monitoring interface of the source database, and the corresponding real-time load indicators are calculated.

[0028] In the embodiments of the present application, the technical indicators are obtained through the database native monitoring interface, and the technical indicators at least include CPU usage, disk I / O throughput, transaction processing delay and buffer pool hit rate. A lightweight probe agent is deployed to collect operating system layer indicators, and the indicators at least include memory usage, network bandwidth utilization and process CPU time. The indicators of the heterogeneous database are standardized, a unified indicator system is constructed, and the same indicators of different databases are mapped to the preset standard indicator dimension.

[0029] Before collecting the real-time load indicators of the source database, a multi-level load threshold matrix is constructed, the configuration center of the source database is analyzed, and a business scenario factor is added.

[0030] Specifically, historical load data of the source database is acquired, a multi-level load threshold matrix is constructed based on the historical load data, the load threshold matrix includes threshold intervals corresponding to each load level, the load level includes light load, standard and heavy load, configuration parameters of the source database are acquired, and the threshold intervals corresponding to each load level are adjusted based on the configuration parameters to update the multi-level load threshold matrix.

[0031] In the embodiments of the present application, explicit business rule input is acquired, the business rule at least includes data synchronization delay threshold, consistency level requirement, and change sensitive field configuration. Historical change log is analyzed, and implicit scenario features are calculated, the features at least include change frequency distribution, change mode identification, and data impact range evaluation. Business scenario feature vector is constructed, the explicit rule is associated with the implicit features to form a multi-dimensional feature descriptor. A multi-level load threshold matrix is preset, the matrix contains technical index threshold intervals corresponding to each load level, index statistical values are calculated using a sliding time window, the statistical values are matched with the threshold matrix to determine the current database load level, and the threshold matrix is dynamically adjusted based on the business scenario feature vector to form business constraint conditions for load level determination.

[0032] S102: calling a change capture engine corresponding to the source database, dynamically adjusting a running mode of the change capture engine according to the real-time load index, and capturing a change event.

[0033] Specifically, the change capture engine corresponding to the source database is called, the change capture engine is a change data collection component pre-set for different database types, the capture strategy is dynamically switched based on the load level, and the data change event in the source database is captured.

[0034] When the source database is a log-type database, the database slave node is disguised or the log stream is acquired through the official log interface, the binary log or the redo log is listened to in real time, the transaction markers in the log are parsed to identify the transaction boundaries, the row-level events containing the operation type and the data before and after the change are extracted, and the log site information is recorded for breakpoint continuation.

[0035] When the source database is a CDC interface-type database, the official CDC interface is called, the incremental data containing the change type identifier is acquired, the incremental pulling is implemented based on the version number or the timestamp returned by the interface, the unstructured data is converted into standardized change events through the metadata mapping module.

[0036] When the source database is a cloud-native database, the change stream service provided by the manufacturer is accessed, the structured change stream containing the shard identifier and the event sequence is acquired, the change stream is consumed in partition through the message queue or the stream processing platform, and the order guarantee of cross-region changes is implemented based on the globally unique sequence number provided by the cloud service.

[0037] When the source database is a non-logging database, a DDL / DML trigger is created in the target table to write the changed data to a shadow table or a memory-optimized table, or a data snapshot is compared periodically to identify changes based on timestamps, version numbers, or hash values, and a soft delete flag is used to capture deletion operations.

[0038] For example, for a MySQL database, as shown in Figure 2 The synchronization service simulates the interaction protocol of MySQL Slava, disguises itself as a MySQL replica by implementing the protocol stack of MySQL Slave, connects to the MySQL server, and sends synchronization commands to the MySQL server after connection. The MySQL server receives the synchronization command sent by the synchronization service, sends the MySQL binlog to the synchronization service according to the MySQL Slave protocol, and captures the database changes by parsing the MySQL binlog sent by the MySQL server. Through the message queue, the database changes are broadcast to the subscribers (modules that need to synchronize heterogeneous data, such as Redis, Elastic Search, etc.), and multiple subscribers can process concurrently. In addition, with the help of the confirmation and consumption compensation mechanism of the message queue, the consistency of distributed transactions can be achieved.

[0039] In the embodiments of the present application, the change pattern recognition step includes: identifying periodic patterns of change frequency based on time series analysis techniques, including daily peak, weekly peak, and monthly peak. An anomaly detection algorithm is applied to identify sudden change events, including at least statistical threshold-based detection and machine learning-based outlier detection. A correlation model between change patterns and business events is established, for example, associating the promotion activities of an e-commerce platform with the surge in order table changes.

[0040] Further, the real-time load indicators are matched in a multi-level load matrix to determine the load level corresponding to the source database.

[0041] When the load level is light, the running mode of the change capture engine is adjusted to full-load running mode, the full-load data operation information of the source database is parsed, and the data change information is extracted; when the load level is standard, the running mode of the change capture engine is adjusted to balanced running mode, the data operation information of the key fields in the source database is obtained and parsed, and the key data change information is extracted; when the load level is heavy, the running mode of the change capture engine is adjusted to local running mode, the data operation information of the key fields is parsed through a multi-row merging algorithm, and the key data change information is extracted.

[0042] In the embodiments of the present application, for a log-type database, in a light-load state, a full-load running mode is adopted, a binary log is completely parsed, line-by-line change events are generated, and the integrity of changed data is ensured; in a standard state, a balanced running mode is adopted, transaction boundaries in the log are parsed, and change sets are generated in batches according to transactions, so as to balance performance and data granularity; and in a heavy-load state, a local running mode is adopted, log events are aggregated based on a time window, and batch operation instructions containing multiple rows of changes are generated, so as to reduce processing overhead.

[0043] For a CDC interface-type database, in a light-load state, a full-load running mode is adopted, and full-load change details are obtained through the CDC interface; in a standard state, a balanced running mode is adopted, and only necessary change fields are obtained; and in a heavy-load state, a local running mode is adopted, and a change merging mechanism is enabled.

[0044] For a cloud-native-type database, in a light-load state, a full-load running mode is adopted, a change stream is subscribed to in real time, and each event is immediately processed; in a standard state, a balanced running mode is adopted, the change stream is processed in batches based on a water line, and a maximum batch processing delay threshold is set; and in a heavy-load state, a local running mode is adopted, consumption parallelism is reduced, message queue buffer depth is increased, and system stability is preferentially ensured.

[0045] For a logless-type database, in a light-load state, a full-load running mode is adopted, each change is captured in real time through a trigger and immediately processed; in a standard state, a balanced running mode is adopted, changes are temporarily stored in an in-memory table, and are submitted in batches according to a fixed time window; and in a heavy-load state, a local running mode is adopted, polling frequency is reduced, and query condition filtering is increased.

[0046] S103: The change event is classified, a corresponding message queue is determined and sent, so as to transmit the change event to a target end through the message queue.

[0047] Specifically, an operation type of the change event is determined, the operation type includes a database definition event and a database operation event, the change event is classified according to the operation type, and is sent to a corresponding message queue.

[0048] When the change event is a database definition event, a structure change label is added to the change event, and is sent to a structure change channel; when the change event is a database operation event, it is judged whether a primary key exists in a change field, if yes, a key update label is added to the change event, and is sent to a strongly consistent message queue, if not, a normal update label is added to the change event, and is sent to an asynchronous message queue.

[0049] In the embodiments of the present application, the operation instruction characteristics of the change event are analyzed, and it is determined whether it belongs to a database definition event (DDL) or a database operation event (DML). The operation corresponding to the DDL event is a database structure definition operation, and the operation corresponding to the DML event is a data addition, deletion or modification operation.

[0050] The business sensitivity of the DML event is further analyzed. When the change involves a primary key field or a sensitive field marked as funds / inventory, it is a key update event; when the change does not contain a primary key and has no non-sensitive field, it is a normal update event; when the change source is an operation log table and has no business consistency requirement, it is a log type event.

[0051] The present application solves the performance rigidity and compatibility problems of traditional data synchronization schemes in a heterogeneous environment through a multi-dimensional dynamic adaptation mechanism.

[0052] Firstly, full-scene coverage of different databases is realized. Through a special change capture engine and a load-aware strategy, data integrity can be ensured when the load is light, and system overhead can be reduced through a merging algorithm and key field filtering when the load is heavy, significantly improving resource utilization.

[0053] Secondly, through semantic event classification and multi-queue transmission mechanism, the system can intelligently select the transmission channel according to the change type and data characteristics, ensure the strong consistency of core data, allow non-critical changes to pass through the asynchronous queue to improve the overall throughput, and achieve the optimal balance between data synchronization quality and system performance.

[0054] As shown in Figure 3 The embodiments of the present application also provide a heterogeneous data synchronization device, which comprises:

[0055] at least one processor; and

[0056] a memory in communication connection with the at least one processor; wherein

[0057] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a heterogeneous data synchronization method as described in any of the above embodiments.

[0058] The embodiments of the present application also provide a non-volatile computer storage medium storing computer executable instructions, which are configured to execute a heterogeneous data synchronization method as described in any of the above embodiments.

[0059] The various embodiments in the present application are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the device and medium embodiments are described simply because they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0060] The device and medium provided by the embodiments of the present application are one-to-one corresponding to the method, and therefore, the device and medium also have the similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be described here.

[0061] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer-usable program code.

[0062] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks. Figure 1 The function specified in one block or multiple blocks.

[0063] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the flowcharts and / or block diagrams. Figure 1 The function specified in one flow or multiple flows and / or blocks. Figure 1 The function specified in one block or multiple blocks.

[0064] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 The flowchart blocks or blocks in the multiple flows and / or blocks

[0065] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0066] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0067] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0068] It should also be noted that the terms "comprising", "including", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0069] ​The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A method of heterogeneous data synchronization, the method comprising: The method comprises the following steps: determining the source database corresponding to the heterogeneous data, collecting real-time load indicators of the source database; the number of source databases is one or more; calling the change capture engine corresponding to the source database, dynamically adjusting the running mode of the change capture engine according to the real-time load indicators, and capturing the change event; classifying the change event, determining the corresponding message queue and sending, and transmitting the change event to the target end through the message queue.

2. The heterogeneous data synchronization method of claim 1, wherein, The types of the source database include log-type database, CDC interface-type database, cloud-native database and non-log-type database; The collection of real-time load indicators of the source database specifically includes: real-time collection of load indicator data through the built-in monitoring interface of the source database, and calculation of the corresponding real-time load indicators.

3. The heterogeneous data synchronization method of claim 2, wherein, Before the determination of the source database corresponding to the heterogeneous data and the collection of real-time load indicators of the source database, the method further comprises: obtaining historical load data of the source database, and constructing a multi-level load threshold matrix based on the historical load data; the load threshold matrix includes threshold intervals corresponding to each load level; the load level includes light load, standard and heavy load; obtaining the configuration parameters of the source database, adjusting the threshold intervals corresponding to each load level based on the configuration parameters, and updating the multi-level load threshold matrix.

4. The heterogeneous data synchronization method of claim 1, wherein, The calling of the change capture engine corresponding to the source database specifically includes: when the source database is a log-type database, a pseudo-slave node of the source database is constructed, a connection with the master library of the source database is established, and a binary log file is received through the pseudo-slave node; when the source database is a CDC interface-type database, a native CDC interface is called to obtain incremental data containing change type identifiers; when the source database is a cloud-native database, a change stream service provided by a cloud database vendor is used to obtain a structured change stream containing shard identifiers; when the source database is a non-log-type database, data snapshots of the source database are compared periodically based on a preset time period.

5. The heterogeneous data synchronization method of claim 3 or claim 4, wherein, The dynamic adjustment of the running mode of the change capture engine according to the real-time load indicators, and the capturing of the change event of the source database specifically include: matching the real-time load indicators in a multi-level load matrix to determine the load level corresponding to the source database; when the load level is light load, the running mode of the change capture engine is adjusted to full-load running mode, full-load data operation information of the source database is parsed, and data change information is extracted; when the load level is standard, the running mode of the change capture engine is adjusted to balanced running mode, data operation information of the key fields in the source database is obtained and parsed, and key data change information is extracted; when the load level is heavy load, the running mode of the change capture engine is adjusted to local running mode, the data operation information of the key fields is parsed through a multi-line merging algorithm, and key data change information is extracted.

6. The heterogeneous data synchronization method of claim 1, wherein, The classification of the change event, the determination of the corresponding message queue and the sending specifically include: determining an operation type of the change event; the operation type comprises a database definition event and a database operation event; classifying the change event according to the operation type and sending to a corresponding message queue.

7. The heterogeneous data synchronization method of claim 6, wherein, The classifying the change event according to the operation type and sending to a corresponding message queue specifically comprises: when the change event is a database definition event, adding a structure change label to the change event and sending to a structure change channel; when the change event is a database operation event, judging whether a primary key exists in a change field; if yes, adding a key update label to the change event and sending to a strong consistency message queue; if no, adding a normal update label to the change event and sending to an asynchronous message queue.

8. The heterogeneous data synchronization method of claim 1, wherein, After the classifying the change event, determining a corresponding message queue and sending, and transmitting the change event to a target end through the message queue, the method further comprises: obtaining a data change execution result of the target end and feeding back the execution result to a monitoring center.

9. A heterogeneous data synchronization device, comprising: comprise: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute as follows: determining a source database corresponding to the heterogeneous data, collecting real-time load indicators of the source database; the number of the source databases is one or more; calling a change capture engine corresponding to the source database, dynamically adjusting a running mode of the change capture engine according to the real-time load indicators, and capturing change events; classifying the change events, determining corresponding message queues and sending, and transmitting the change events to target ends through the message queues.

10. A non-transitory computer storage medium storing computer-executable instructions that, when executed, cause a computer to perform: The computer executable instructions are configured to execute as follows: determining a source database corresponding to the heterogeneous data, collecting real-time load indicators of the source database; the number of the source databases is one or more; calling a change capture engine corresponding to the source database, dynamically adjusting a running mode of the change capture engine according to the real-time load indicators, and capturing change events; classifying the change events, determining corresponding message queues and sending, and transmitting the change events to target ends through the message queues.

Images