Data synchronization system, method and equipment between heterogeneous databases and medium

Abstract

The invention relates to the field of computer software, and discloses a data synchronization system, method and equipment between heterogeneous databases and a medium. The system comprises a source end subsystem used for acquiring to-be-synchronized data in a source end heterogeneous database and writing the to-be-synchronized data into a preset data transmission summary table; the source terminal subsystem is also used for responding to the data transmission task, determining target synchronous data from the data transmission summary table, and sending the target synchronous data to the target terminal subsystem in batches; the target terminal subsystem is used for receiving the target synchronization data sent by the source terminal subsystem, verifying the target synchronization data, calculating a corresponding target sub-library sub-table based on a preset service key, writing the verified target synchronization data into the target sub-library sub-table of the target terminal heterogeneous database, and generating a data processing result; and the target terminal subsystem is also used for feeding back the data processing result to the source terminal subsystem. According to the method, the efficiency, reliability and operation and maintenance convenience of data synchronization between heterogeneous databases are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer software, and in particular to a data synchronization system, method, device and medium for heterogeneous databases. Background Technology

[0002] During enterprise digital transformation, due to historical reasons, business needs, or differences in technology selection, different business systems often adopt heterogeneous database architectures. For example, an older sales personnel management system (hereinafter referred to as System A) may use a monolithic Oracle database, while a new generation of big data analytics platform or distributed CRM system (hereinafter referred to as System B) may use a distributed OceanBase database. To support business collaboration, System A needs to synchronize key sales data to System B in a timely manner so that System B can perform unified data analysis and business expansion.

[0003] In existing technologies, the following methods are often used: (1) Data in the database of system A is converted into a format and then written to the database of system B in batches using an ETL tool; (2) Triggers are created on the tables of the database of system A, and when data is added, deleted, or modified, they are directly connected to the database of system B and the corresponding write operation is performed; (3) System A publishes data change events to a message queue in its business code, and system B pulls data from the queue and writes it to its local database. However, the above methods have problems such as poor real-time performance, high coupling, difficulty in distributed adaptation, and invisible state. Therefore, there is an urgent need for a technical solution that can take into account real-time performance, reliability, accuracy, and traceability. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a data synchronization system, method, device and medium for heterogeneous databases.

[0005] This invention provides the following technical solution: In a first aspect, the present invention provides a data synchronization system between heterogeneous databases. The system includes a source subsystem and a target subsystem. The source subsystem and the target subsystem are communicatively connected. The source subsystem is connected to a source heterogeneous database, and the target subsystem is connected to a target heterogeneous database. The source heterogeneous database and the target heterogeneous database are databases of different types. The source terminal system is used to acquire data to be synchronized from the source heterogeneous database and write the data to be synchronized into a preset data transmission summary table. The source terminal system is also configured to, in response to a data transmission task, determine target synchronization data from the data transmission summary table and send the target synchronization data in batches to the target terminal system; The target terminal system is used to receive target synchronization data sent by the source terminal system, verify the target synchronization data, calculate the corresponding target database sharding and table partitioning based on the preset business key, write the verified target synchronization data into the target database sharding and table partitioning of the target terminal heterogeneous database, and generate data processing results. The target terminal system is also used to feed back the data processing results to the source terminal system.

[0006] In an optional implementation, the source terminal system includes a data acquisition module and a summary table writing module; The data acquisition module is used to acquire business data changes in the heterogeneous database at the source end in real time, and extract the data to be synchronized from the business data changes. The business data changes include adding business data, modifying business data, and deleting business data. The summary table writing module is used to write the data to be synchronized into the preset data transmission summary table according to a preset unified format, and to configure a status identifier for each piece of data to be synchronized, the status identifier including not transmitted, transmitting, transmission successful, and transmission failed.

[0007] In an optional implementation, the source terminal system includes a task triggering module and a data reading module; The task triggering module is used to start the data transmission task according to preset rules in a dual-trigger mode. The dual-trigger mode includes a timed scheduling trigger mode and a business event-driven trigger mode. The data reading module is used to read the data to be synchronized that is marked as not transmitted from the preset data transmission summary table after the data transmission task is started, determine it as the target synchronization data, and use a preset paging row lock mechanism to control the number of target synchronization data read.

[0008] In an optional implementation, the source terminal system further includes an interface calling module and a status update module; The interface calling module is used to package the target synchronization data into batch data, call the preset data receiving interface of the target subsystem through a preset client tool, and send the batch data to the target subsystem through the preset data receiving interface; The status update module is used to update the status identifier of the target synchronization data to "transmitting" in the preset data transmission summary table after the data reading module reads the target synchronization data, and to update the status identifier of the target synchronization data according to the data processing result after receiving the data processing result from the target terminal system.

[0009] In an optional implementation, the target terminal system includes an interface receiving module and a data verification module; The interface receiving module is used to receive batch data sent by the source subsystem through the preset data receiving interface, parse the format of the batch data, and extract the complete information of each target synchronization data in the batch data; The data verification module is used to perform legality verification on each parsed target synchronization data, generate verification results, filter out target synchronization data whose verification results are passed, and record target synchronization data that fail verification and the specific reasons for failure. The legality verification includes verification of the integrity of key business fields, verification of data type legality, and verification of data format standardization.

[0010] In an optional implementation, the target terminal system further includes a routing calculation module, a distributed writing module, and a result feedback module; The routing calculation module is used to calculate the target database sharding and table sharding for each piece of the verified target synchronization data based on a preset business key and using a preset sharding algorithm. This sharding is used as the storage location for each piece of the verified target synchronization data in the heterogeneous database on the target end. The preset sharding algorithm includes a consistent hashing algorithm and a modulo algorithm. The distributed write module is used to use a database sharding middleware or a dedicated driver for the heterogeneous database on the target end to batch write each of the verified target synchronization data to the corresponding target database sharding and table, generate data write results, and generate data processing results based on the data write results and the verification results. The data processing results include the transmission status and failure reason of each of the target synchronization data, and the transmission status includes transmission success and transmission failure. The result feedback module is used to feed back the data processing result to the source terminal system through the preset data receiving interface.

[0011] In an optional implementation, the source subsystem further includes a failure retry module, and the preset data transmission summary table is configured with a retry counter field for recording the number of retries of the data to be synchronized; The failure retry module is used to periodically scan the data to be synchronized in the preset data transmission summary table whose status is marked as transmission failure and whose value of the retry counter field has not reached the preset retry threshold. If the data is identified as transmission failure data, the task triggering module is triggered to restart the data transmission task for the transmission failure data, and the value of the retry counter field of the corresponding transmission failure data is incremented by 1 before each retry. The failure retry module is also used to periodically scan the data to be synchronized that has a status of transmission failure and the value of the retry counter field reaches the preset retry threshold, determine it as retry failure data, generate an alarm notification, send the alarm notification to relevant personnel, and record the identification information, failure reason and retry count of the retry failure data in a preset log table.

[0012] Secondly, the present invention provides a data synchronization method between heterogeneous databases, applied to a data synchronization system between heterogeneous databases as described in the first aspect. The system includes a source subsystem and a target subsystem, the source subsystem and the target subsystem being communicatively connected. The source subsystem is connected to a corresponding source heterogeneous database, and the target subsystem is connected to a corresponding target heterogeneous database. The source heterogeneous database and the target heterogeneous database are databases of different types. The method includes: The source terminal subsystem obtains the data to be synchronized from the heterogeneous database of the source terminal, and writes the data to be synchronized into a preset data transmission summary table; In response to a data transmission task, the source terminal system determines the target synchronization data from the data transmission summary table and sends the target synchronization data in batches to the target terminal system. The target terminal system receives target synchronization data sent by the source terminal system, verifies the target synchronization data, calculates the corresponding target database sharding and table partitioning based on the preset business key, writes the verified target synchronization data into the target database sharding and table partitioning of the heterogeneous database on the target terminal, and generates data processing results. The data processing results are fed back to the source terminal system through the target terminal system.

[0013] Thirdly, this disclosure provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the data synchronization method between heterogeneous databases described in the second aspect.

[0014] Fourthly, this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data synchronization method between heterogeneous databases described in the second aspect.

[0015] The beneficial effects of this application are: The data synchronization system between heterogeneous databases provided in this application embodiment constructs a "data transmission summary table" in the source system as a state buffer center. Combined with a timed / event-triggered batch interface call mechanism, it achieves reliable data push to the target system in the heterogeneous database. A distributed routing strategy based on business keys is introduced on the target end to ensure data accurately lands in the correct database shards and tables. Simultaneously, a state feedback mechanism enables end-to-end visibility of data synchronization status and anomaly tracing. This achieves timely, reliable, and accurate data synchronization between heterogeneous database systems, while possessing characteristics of low coupling, traceability, and easy scalability. It is comprehensively adaptable to enterprise-level cross-system scenarios such as life insurance sales personnel management systems, significantly improving the efficiency, reliability, and ease of operation and maintenance of data synchronization.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the various drawings, similar components are numbered similarly.

[0018] Figure 1 This illustration shows a schematic diagram of the structure of a data synchronization system between heterogeneous databases provided in an embodiment of this application; Figure 2 A flowchart of a data synchronization method between heterogeneous databases provided in an embodiment of this application is shown; Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of this application is shown.

[0019] Explanation of key component symbols: 100 - Data synchronization system between heterogeneous databases; 110 - Source subsystem; 120 - Target subsystem; 111 - Data acquisition module; 112 - Summary table writing module; 113 - Task triggering module; 114 - Data reading module; 115 - Interface call module; 116 - Status update module; 117 - Failure retry module; 121 - Interface receiving module; 122 - Data verification module; 123 - Route calculation module; 124 - Distributed writing module; 125 - Result feedback module. Detailed Implementation

[0020] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0021] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the template description is for the purpose of describing particular embodiments only and is not intended to limit the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0023] Example 1 like Figure 1 The diagram shown is a structural schematic of a data synchronization system 100 between heterogeneous databases according to an embodiment of this application. The system includes a source subsystem 110 and a target subsystem 120. The source subsystem 110 and the target subsystem 120 establish a bidirectional communication connection through a network, enabling batch data transmission and processing result feedback. The source subsystem 110 is connected to a source heterogeneous database (such as an Oracle monolithic relational database), and the target subsystem 120 is connected to a target heterogeneous database (such as an OceanBase distributed database). The source heterogeneous database and the target heterogeneous database are different types of databases, which can adapt to the typical heterogeneous architecture scenario of "old system monolithic database - new system distributed database" in enterprises.

[0024] Preferably, the source subsystem 110 is used to acquire the data to be synchronized in the heterogeneous database of the source end and write the data to be synchronized into a preset data transmission summary table; the source subsystem 110 is also used to respond to the data transmission task, determine the target synchronization data from the data transmission summary table, and send the target synchronization data in batches to the target subsystem 120.

[0025] Understandably, the source subsystem 110 is developed based on the Java language. Its core is used to complete the collection, buffer storage and transmission triggering of data to be synchronized in the heterogeneous database of the source end. Specifically, it includes a data collection module 111, a summary table writing module 112, a task triggering module 113, a data reading module 114, an interface call module 115 and a status update module 116.

[0026] Optionally, the data acquisition module 111 is used to intercept the business Service methods of the source subsystem 110 using Spring AOP (Aspect-Oriented Programming) technology, and obtain business data changes in the heterogeneous database of the source end in real time, including business data addition, business data modification and business data deletion; and extract key business data (such as customer ID, transaction record ID, contract amount, sales ID, etc.) from the business data changes as data to be synchronized, thereby decoupling data synchronization from the main business, avoiding direct operation on the core business tables of the source end, and reducing the coupling with the main business process.

[0027] Optionally, the summary table writing module 112 is used to perform batch insert operations through the MyBatis persistence layer framework, write the data to be synchronized into a preset data transmission summary table in a preset unified format (such as JSON format), and configure a status flag for each piece of data to be synchronized. The status flag includes not transmitted, in transmission, transmission successful, and transmission failed. The data lifecycle is precisely controlled through the status bits to ensure that each piece of data can be correctly processed or traced, which solves the problem of the status not being visible in the existing solution.

[0028] Optionally, the task triggering module 113 is used to start data transmission tasks according to preset rules using a dual-trigger mode. The dual-trigger mode includes a timed scheduling trigger mode and a business event-driven trigger mode. The timed scheduling trigger is implemented based on the Spring Scheduler component and can be triggered at preset intervals (such as every 30 seconds) to adapt to common data synchronization scenarios. The business event-driven trigger is implemented based on an event listener and can respond to key business events on the source side (such as sales order completion events and monthly performance statistics start events) to trigger real-time data transmission and meet real-time requirements (such as real-time sales data dashboard updates). Compared with direct connection to triggers, this avoids strong coupling to the target subsystem and improves the overall stability and throughput of the system.

[0029] The data reading module 114 is used to read data marked as "not transmitted" from a preset data transmission summary table after the data transmission task is started by the task triggering module 113. This data is then identified as the target synchronization data. To prevent data from being read repeatedly or tampered with in concurrent scenarios, the data reading module 114 uses a "pagination query + row lock mechanism" to control the number of target synchronization data reads. Specifically, a fixed statement can be executed to read up to 100 data entries at a time (this can be adjusted according to system performance), and row locks are used to lock the read data to prevent other threads from processing it repeatedly.

[0030] Optionally, the interface call module 115 is used to package the target synchronization data into batch data, and call the preset data receiving interface of the target subsystem 120 through a preset client tool such as RestTemplate or FeignClient to send the batch data to the target subsystem 120. This interface call module 115 supports batch transmission instead of single transmission, which can significantly improve the data synchronization throughput and adapt to the large-scale data synchronization scenario of the source subsystem 110.

[0031] Optionally, the status update module 116 is used to immediately update the status identifier of the target synchronization data to "transmission in progress" in the preset data transmission summary table after the data reading module 114 reads the target synchronization data, so as to prevent the data from being repeatedly scheduled; and after receiving the data processing result fed back by the target terminal system 120, update the status identifier of the target synchronization data according to the data processing result. If the processing result is "transmission successful", it is updated to "transmission successful"; if the processing result is "transmission failed", it is updated to "transmission failed".

[0032] Preferably, the target terminal system 120 is used to receive target synchronization data sent by the source terminal system 110, verify the target synchronization data, calculate the corresponding target sharding and table based on the preset business key, write the verified target synchronization data into the target sharding and table of the heterogeneous database of the target terminal, and generate data processing results; the target terminal system 120 is also used to feed back the data processing results to the source terminal system 110.

[0033] Understandably, the target terminal system 120 is also developed based on the Java language. Its core is used to complete the reception, verification, distributed routing and writing of batch data from the source end. Specifically, it includes an interface receiving module 121, a data verification module 122, a route calculation module 123, a distributed writing module 124 and a result feedback module 125.

[0034] Optionally, the interface receiving module 121 is used to receive batch data sent by the source subsystem 110 through a preset data receiving interface. The Spring MVC Controller component can be used to parse the batch data, extract the complete information of each target synchronized data in the batch data (such as customer ID, transaction amount, sales ID, etc.), and perform preliminary filtering on the data with abnormal format (such as JSON syntax errors, missing fields) that occur during the parsing process to prevent invalid data from entering the subsequent process.

[0035] Optionally, the data verification module 122 is used to perform legality verification on each parsed target synchronization data. The verification content specifically includes: (1) integrity verification of key business fields (such as whether business keys such as customer ID and sales ID are missing, if missing, it is determined to be a verification failure); (2) legality verification of data types (such as whether the "contract amount" field is a numeric type and whether the "transaction date" field is a date format, which must meet the field type requirements of the heterogeneous database on the target side); (3) data format standardization verification (such as whether the customer ID meets the preset format of "prefix + number"); after the verification is completed, the verification result is generated, the target synchronization data with the verification result passed is filtered out, and the target synchronization data that failed the verification and the specific reason for failure (such as "contract amount is a string type, which does not meet the numeric requirements") are recorded, so as to provide a basis for the generation of subsequent data processing results.

[0036] Optionally, the routing calculation module 123 is used to calculate the target database sharding and table partitioning for each piece of verified target synchronized data, based on a preset business key (such as customer ID, sales ID, which can be configured according to business scenarios) and a preset sharding algorithm. This sharding algorithm serves as the storage location for each piece of verified target synchronized data in the target heterogeneous database. The preset sharding algorithm includes a consistent hashing algorithm (suitable for scenarios with dynamic expansion of target database sharding and table partitioning) and a modulo algorithm (suitable for scenarios with a fixed number of database sharding and table partitioning). For example, using customer ID as the business key, the consistent hashing algorithm calculates that the customer data should be written to the "order_12" partitioning table in the "customer_05" database, accurately determining the storage location of the data in the target heterogeneous database and solving the problem of difficult distributed deployment adaptation.

[0037] Optionally, the distributed write module 124 is used to use a database sharding middleware (such as ShardingSphere-JDBC) or a dedicated driver for the target heterogeneous database (such as OceanBase dedicated JDBC driver) to batch write each verified target synchronization data to the corresponding target sharded database and table. During the write process, transactional batch operations are supported to ensure that data in the same batch is either all written successfully or all rolled back, avoiding data inconsistency caused by partial data write failure. At the same time, the distributed write module 124 generates data write results (each data is "written successfully" or "written failed", with failure reasons such as "target sharded table storage node crashed" or "insufficient permissions"), and combines them with the verification results of the data verification module 122 to integrate and generate a complete data processing result. The data processing result includes the transmission status (transmission successful / transmission failed) and failure reason (verification failure reason or write failure reason) of each target synchronization data.

[0038] Optionally, the result feedback module 125 is used to feed back the data processing results to the source subsystem 110 through a preset data receiving interface. In this embodiment, the feedback content can be organized in a hierarchical manner of "batch data - single detail", including not only the total number of batch data, the number of successful data, and the number of failed data, but also the specific identifier and reason for each failed data. This ensures that the source subsystem 110 can accurately update the status identifier of the corresponding data, achieving traceability and high reliability of data synchronization, and significantly reducing the cost of manual intervention.

[0039] In an optional implementation, the source subsystem 110 further includes a failure retry module 117, which has a preset data transmission summary table configured with a retry counter field for recording the number of retries of the data to be synchronized. This can subsequently record the number of retries after data synchronization failure, providing a data basis for status tracking and retry scheduling.

[0040] Understandably, the failure retry module 117 scans the preset data transmission summary table at regular intervals (e.g., every 1 minute) and processes two types of data separately: (1) For data to be synchronized with a status indicating transmission failure and the value of the retry counter field not reaching the preset retry threshold (3-8 times, configurable as needed), it is determined to be transmission failure data, triggering the task triggering module 113 to restart the data transmission task for the transmission failure data, and incrementing the value of the retry counter field of the corresponding transmission failure data by 1 before each retry, which can effectively deal with temporary faults such as network jitter and short-term downtime of the target end; (2) For data to be synchronized with a status indicating transmission failure and the value of the retry counter field reaching the preset retry threshold, it is determined to be retry failure data, generating an alarm notification, which is sent to maintenance or business-related personnel through an instant messaging robot (e.g., WeChat robot, DingTalk robot), and at the same time, the identification information, failure reason and retry count of the retry failure data are recorded in the preset log table, which improves the data synchronization success rate and facilitates problem tracing and manual intervention.

[0041] The data synchronization system between heterogeneous databases provided in this application embodiment constructs a "data transmission summary table" in the source system as a state buffer center. Combined with a timed / event-triggered batch interface call mechanism, it achieves reliable data push to the target system in the heterogeneous database. A distributed routing strategy based on business keys is introduced on the target end to ensure data accurately lands in the correct database shards and tables. Simultaneously, a state feedback mechanism enables end-to-end visibility of data synchronization status and anomaly tracing. This achieves timely, reliable, and accurate data synchronization between heterogeneous database systems, while possessing characteristics of low coupling, traceability, and easy scalability. It is comprehensively adaptable to enterprise-level cross-system scenarios such as life insurance sales personnel management systems, significantly improving the efficiency, reliability, and ease of operation and maintenance of data synchronization.

[0042] Example 2 like Figure 2 The diagram shows a flowchart of a data synchronization method between heterogeneous databases according to an embodiment of this application. The data synchronization method provided in this embodiment is applied to a data synchronization system between heterogeneous databases as described in Embodiment 1. The system includes a source subsystem and a target subsystem, which are communicatively connected. The source subsystem connects to a corresponding heterogeneous database, and the target subsystem connects to a corresponding heterogeneous database. The source and target heterogeneous databases are different types of databases. The method specifically includes the following steps: Step S110: Obtain the data to be synchronized from the heterogeneous database of the source terminal through the source terminal subsystem, and write the data to be synchronized into the preset data transmission summary table. Step S120: In response to the data transmission task, the source terminal system determines the target synchronization data from the data transmission summary table and sends the target synchronization data to the target terminal system in batches. Step S130: Receive target synchronization data sent by the source terminal system through the target terminal system, verify the target synchronization data, calculate the corresponding target database sharding and table partitioning based on the preset business key, write the verified target synchronization data into the target database sharding and table partitioning of the heterogeneous database on the target terminal, and generate data processing results. Step S140: The data processing results are fed back to the source terminal system through the target terminal system.

[0043] The data synchronization method between heterogeneous databases provided in this application embodiment can realize all processes of the data synchronization system between heterogeneous databases corresponding to Embodiment 1, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0044] Example 3 This application also provides a computer device. Please refer to the following for details. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0045] The computer device 3 includes a memory 31, a processor 32, and a network interface 33 that are interconnected via a system bus. It should be noted that only a computer device 3 with a memory 31, a processor 32, and a network interface 33 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0046] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0047] The memory 31 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D slot compatibility test memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 31 may be an internal storage unit of the computer device 3, such as the hard disk or memory of the computer device 3. In other embodiments, the memory 31 may also be an external storage device of the computer device 3, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 3. Of course, the memory 31 may also include both the internal storage unit and its external storage device of the computer device 3. In this embodiment, the memory 31 is typically used to store the operating system and various application software installed on the computer device 3, such as computer-readable instructions for slot compatibility testing methods. In addition, the memory 31 can also be used to temporarily store various types of data that have been output or will be output.

[0048] In some embodiments, the processor 32 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data synchronization chip for heterogeneous databases. The processor 32 is typically used to control the overall operation of the computer device 3. In this embodiment, the processor 32 is used to execute computer-readable instructions stored in the memory 31 or to process data, such as executing computer-readable instructions for the slot compatibility testing method.

[0049] The network interface 33 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 3 and other electronic devices.

[0050] The computer device provided in this embodiment can execute the above-described data synchronization method between heterogeneous databases. Here, the data synchronization method between heterogeneous databases can be the same as the data synchronization method between heterogeneous databases described in the various embodiments above.

[0051] Example 4 This embodiment also provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the data synchronization method between heterogeneous databases in this embodiment.

[0052] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0053] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can also be implemented in other ways. The system embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that, as an alternative implementation, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0054] In addition, the functional modules or units in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0055] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium can be a non-volatile storage medium or a volatile storage medium. For example, the storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code.

[0056] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A data synchronization system between heterogeneous databases, characterized in that, The system includes a source terminal system and a target terminal system. The source terminal system and the target terminal system are communicatively connected. The source terminal system is connected to a source heterogeneous database, and the target terminal system is connected to a target heterogeneous database. The source heterogeneous database and the target heterogeneous database are databases of different types. The source terminal system is used to acquire data to be synchronized from the source heterogeneous database and write the data to be synchronized into a preset data transmission summary table. The source terminal system is also configured to, in response to a data transmission task, determine target synchronization data from the data transmission summary table and send the target synchronization data in batches to the target terminal system; The target terminal system is used to receive target synchronization data sent by the source terminal system, verify the target synchronization data, calculate the corresponding target database sharding and table partitioning based on the preset business key, write the verified target synchronization data into the target database sharding and table partitioning of the target terminal heterogeneous database, and generate data processing results. The target terminal system is also used to feed back the data processing results to the source terminal system.

2. The data synchronization system between heterogeneous databases according to claim 1, characterized in that, The source terminal system includes a data acquisition module and a summary table writing module; The data acquisition module is used to acquire business data changes in the heterogeneous database at the source end in real time, and extract the data to be synchronized from the business data changes. The business data changes include adding business data, modifying business data, and deleting business data. The summary table writing module is used to write the data to be synchronized into the preset data transmission summary table according to a preset unified format, and to configure a status identifier for each piece of data to be synchronized, the status identifier including not transmitted, transmitting, transmission successful, and transmission failed.

3. The data synchronization system between heterogeneous databases according to claim 2, characterized in that, The source terminal system includes a task triggering module and a data reading module; The task triggering module is used to start the data transmission task according to preset rules in a dual-trigger mode. The dual-trigger mode includes a timed scheduling trigger mode and a business event-driven trigger mode. The data reading module is used to read the data to be synchronized that is marked as not transmitted from the preset data transmission summary table after the data transmission task is started, determine it as the target synchronization data, and use a preset paging row lock mechanism to control the number of target synchronization data read.

4. The data synchronization system between heterogeneous databases according to claim 3, characterized in that, The source terminal system also includes an interface call module and a status update module; The interface calling module is used to package the target synchronization data into batch data, call the preset data receiving interface of the target subsystem through a preset client tool, and send the batch data to the target subsystem through the preset data receiving interface; The status update module is used to update the status identifier of the target synchronization data to "transmitting" in the preset data transmission summary table after the data reading module reads the target synchronization data, and to update the status identifier of the target synchronization data according to the data processing result after receiving the data processing result from the target terminal system.

5. The data synchronization system between heterogeneous databases according to claim 4, characterized in that, The target terminal system includes an interface receiving module and a data verification module; The interface receiving module is used to receive batch data sent by the source subsystem through the preset data receiving interface, parse the format of the batch data, and extract the complete information of each target synchronization data in the batch data; The data verification module is used to perform legality verification on each parsed target synchronization data, generate verification results, filter out target synchronization data whose verification results are passed, and record target synchronization data that fail verification and the specific reasons for failure. The legality verification includes verification of the integrity of key business fields, verification of data type legality, and verification of data format standardization.

6. The data synchronization system between heterogeneous databases according to claim 5, characterized in that, The target terminal system also includes a routing calculation module, a distributed writing module, and a result feedback module; The routing calculation module is used to calculate the target database sharding and table sharding for each piece of the verified target synchronization data based on a preset business key and using a preset sharding algorithm. This sharding is used as the storage location for each piece of the verified target synchronization data in the heterogeneous database on the target end. The preset sharding algorithm includes a consistent hashing algorithm and a modulo algorithm. The distributed write module is used to use a database sharding middleware or a dedicated driver for the heterogeneous database on the target end to batch write each of the verified target synchronization data to the corresponding target database sharding and table, generate data write results, and generate data processing results based on the data write results and the verification results. The data processing results include the transmission status and failure reason of each of the target synchronization data, and the transmission status includes transmission success and transmission failure. The result feedback module is used to feed back the data processing result to the source terminal system through the preset data receiving interface.

7. The data synchronization system between heterogeneous databases according to claim 6, characterized in that, The source terminal system also includes a failure retry module, and the preset data transmission summary table is configured with a retry counter field for recording the number of retries of the data to be synchronized; The failure retry module is used to periodically scan the data to be synchronized in the preset data transmission summary table whose status is marked as transmission failure and whose value of the retry counter field has not reached the preset retry threshold. If the data is identified as transmission failure data, the task triggering module is triggered to restart the data transmission task for the transmission failure data, and the value of the retry counter field of the corresponding transmission failure data is incremented by 1 before each retry. The failure retry module is also used to periodically scan the data to be synchronized that has a status of transmission failure and the value of the retry counter field reaches the preset retry threshold, determine it as retry failure data, generate an alarm notification, send the alarm notification to relevant personnel, and record the identification information, failure reason and retry count of the retry failure data in a preset log table.

8. A method for data synchronization between heterogeneous databases, characterized in that, A data synchronization system for heterogeneous databases as described in any one of claims 1-7, the system comprising a source subsystem and a target subsystem, the source subsystem and the target subsystem being communicatively connected, the source subsystem correspondingly accessing a source heterogeneous database, and the target subsystem correspondingly accessing a target heterogeneous database, wherein the source heterogeneous database and the target heterogeneous database are databases of different types, the method comprising: The source terminal subsystem obtains the data to be synchronized from the heterogeneous database of the source terminal, and writes the data to be synchronized into a preset data transmission summary table; In response to a data transmission task, the source terminal system determines the target synchronization data from the data transmission summary table and sends the target synchronization data in batches to the target terminal system. The target terminal system receives target synchronization data sent by the source terminal system, verifies the target synchronization data, calculates the corresponding target database sharding and table partitioning based on the preset business key, writes the verified target synchronization data into the target database sharding and table partitioning of the heterogeneous database on the target terminal, and generates data processing results. The data processing results are fed back to the source terminal system through the target terminal system.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the data synchronization method between heterogeneous databases as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data synchronization method between heterogeneous databases as described in claim 8.

Images