Data stream type synchronization system, device and method

Abstract

A data stream type synchronization system comprises a synchronization configuration unit, a message reading unit, a data processing unit and a data storage unit. The synchronization configuration unit is configured to maintain incremental subscription information of a target system, and determine whether to start increment synchronization of a related table, a field range needing synchronization, synchronization table information written by a target end and audit table information selected by the target end according to subscription configuration; the message reading unit is configured to perform format check and exception handling on incremental messages; the data processing unit is configured to perform parsing and processing of a message structure; and the data storage unit is configured to execute filtering and grouping of the incremental messages, matching and sorting of message major keys, insertion of the incremental data, submission and rollback of database transactions and exception capture and throwing, wherein all units of the data stream type synchronization system form a star topology structure, and data synchronization association between the upstream and downstream systems is decoupled based on the message queue middleware.

Description

Technical field[0001]The present invention relates to the field of computer, and more particularly to a data stream synchronization system, apparatus, and method.Background technique[0002]As competition between enterprises in various industries is increasingly fierce, data informationization is one of the necessary paths of enterprises to improve industry competitiveness. With the development of various enterprise data informationization, more and more information systems have greatly improved the information level of enterprises. However, due to the same architecture of various systems, the application often faces information from information island. The function of the system and the system is repeated, collaborative, lack of norms, interaction complex, resulting in a single system if you need to open service to other system or data interaction, requires a system open interface for multiple different architectures, and even partial business systems The status quo of database envir...

AI Technical Summary

Problems solved by technology

However, due to the different architectures of the early systems and the lack of long-term planning, applications often face the problem of information islands

Functional duplication between systems, mutual collaboration lacks norms, and interaction is complex. As a result, if a single system needs to open services to other systems or perform data interaction, it needs to open interfaces to multiple systems with different architectures, and even some business systems. Differences in database environments between databases, inconsistent data formats and other system statuses lead to blockage of data transmission channels

At the same time, with the development of business, more and more branches are spread all over the country and around the world. The large geographical span and complex network environment have also caused obstacles to information exchange.

[0003] Therefore, interconnection, information sharing, and business collaboration among various business systems are urgent problems to be solved for the development of enterprises in various industries. The key to solving these problems lies in how to build a unified data integration platform or data lake. However, after the platform is established, how to Meeting the functional requirements and timeliness requirements of downstream systems for data integration and synchronization is a relatively complex issue

Images