A data integration system

Abstract

This application relates to a data integration system. The system includes: a source data module, a metadata module, a stream computing module, a data warehouse module, and a batch computing module. The source data module is the source of the integrated data, including relational databases, non-relational databases, and data warehouses. The metadata module extracts metadata from the data sources in the source data module and synchronizes, manages, and stores the metadata, providing a visual management and operation platform. The stream computing module integrates the data sources and metadata in the form of stream computing, synchronizing the data and metadata from the data sources to the data warehouse module. The data warehouse module serves as a transit point and destination for data integration, storing data containing data sources and metadata or distributing data to downstream systems. The batch computing module performs table maintenance tasks, table renaming tasks, and table tracing tasks on multiple data sources within the data warehouse module. This system enables data integration.

Description

Technical Field

[0001] This application relates to the field of data integration technology, and in particular to a data integration system. Background Technology

[0002] With the development of big data technology, more and more data can be obtained from the Internet and other networks through portals, programmable interfaces, etc. Internet-based databases and data integration are becoming increasingly important, with databases storing data of various types and dimensions. However, due to the heterogeneity of data formats, structures, and semantics, it is difficult to acquire and integrate diverse data. Currently, a large amount of data is too scattered, inconvenient to use, and inefficient in data extraction and utilization. Summary of the Invention

[0003] Therefore, it is necessary to provide a data integration system that can achieve data integration to address the aforementioned technical problems.

[0004] A data integration system, the system comprising: a source data module, a metadata module, a stream computing module, a data warehouse module, and a batch computing module;

[0005] The source data module is the source of the integrated data, including relational databases, non-relational databases, and data warehouses; relational databases, non-relational databases, and data warehouses include one or more data sources;

[0006] The metadata module is used to extract metadata from the data source of the source data module, synchronize, manage and store the metadata, and provide a visual management and operation platform;

[0007] The stream computing module is used to integrate data sources and metadata in the form of stream computing, and to synchronize data and metadata from the data source to the data warehouse module;

[0008] The data warehouse module serves as a transit point and destination for data integration, used to store data containing data sources and metadata or to distribute data to downstream systems.

[0009] The batch processing module is used in the data warehouse module to perform table maintenance tasks, table renaming tasks, and table tracing tasks on multiple data sources.

[0010] In one embodiment, the metadata module includes a metadata storage module, a metadata synchronization module, and a metadata management platform. The metadata storage module stores metadata from the data source. The metadata synchronization module provides multiple data flows for other components to call. The data flows include accessing the data source, data source synchronization, data processing, table renaming, table tracking, and table maintenance. The metadata management platform synchronizes metadata from the data source to the metadata module and provides functions for managing the metadata in the metadata storage module, including data dashboards, data maps, data lineage, and data tags, for defining data sources and performing data source synchronization.

[0011] In one embodiment, accessing the data source means pulling all dataset information from the data source and synchronously specifying the dataset to be accessed to the metadata storage, creating a data stream and initializing the state information to the data stream state storage module, creating a log table for the data source in the data warehouse module, creating a data table for each dataset according to the metadata, creating a corresponding log / data message queue, and finally creating and starting a pair of data persistence tasks and data ingestion tasks to start data integration.

[0012] Data source synchronization means traversing the specified dataset in the data source to synchronize the metadata in the dataset to the metadata module storage; the metadata includes the dataset's fields, partitions, and constraint information.

[0013] In one embodiment, the performance indicates that new metadata is synchronized and obtained from the data source, a new version of the data table is created in the data warehouse module, and a corresponding log / data message queue is created. The version and table configuration of the data stream are modified in the data stream state storage module to use the new version of the data table. After the data persistence task finishes consuming the message queue corresponding to the old table, the data persistence task is restarted to start consuming the new table messages from the message queue corresponding to the new table.

[0014] In one embodiment, table renaming means waiting for the data ingestion task to complete its last checkpoint, then stopping the task; synchronizing and obtaining new metadata from the data source to create a new version of the data table in the data warehouse module; simultaneously creating a corresponding log / data message queue; waiting for the data persistence task to write all records of the temporary table in the log message queue to the temporary table partition of the data warehouse module, then starting the table renaming task, sending deletion messages to the message queues corresponding to the old table and log table for records that no longer exist in the temporary table, and sending addition messages to the message queues corresponding to the new table and log table for records of the temporary table; waiting for the data persistence task to consume all messages, then stopping the data persistence task, configuring the version and table of the corresponding data stream in the data stream state storage module to use the new version of the data table and resetting the state; finally starting the data persistence task and the data ingestion task to begin consuming the new version of the message queue.

[0015] In one embodiment, table maintenance refers to querying and traversing the database of the data warehouse module from the metadata module, stopping the corresponding data persistence task and data ingestion task and their monitoring, starting the table maintenance task for the data source and waiting for it to complete, and then restoring the corresponding data persistence task and data ingestion task and their monitoring.

[0016] In one embodiment, the stream computing module includes a data stream state storage module, a data persistence task, and a data ingestion task; each data source corresponds to a data stream, and the data stream includes a data persistence task and a data ingestion task; the data stream is stored in the data stream state storage module; the stream computing module is used to integrate data from the data source and metadata in the form of stream computing.

[0017] In one embodiment, the data ingestion task involves calling the metadata synchronization module to retrieve metadata for all datasets from a specified data source, which is then used to parse data records. Changes to data and metadata are captured from the specified data source using a change data capture method, and format conversion and data integration are performed. The integrated data records are then sent to the data persistence task for processing via a message queue. The sent data records contain both data and logs, using two separate message queues. When the data stream state changes, the data stream state is synchronized to the data stream state storage. In the performance-driven mode, the latest table structure is parsed from the data records to determine if a performance-driven process has occurred. If so, data stream processing is paused, and the metadata synchronization module is called to initiate the performance-driven process. After the performance-driven process of the metadata synchronization module is completed, the data stream state and table structure are updated before normal record processing resumes. In the table renaming mode, the renaming pattern is identified from the data records, and the metadata synchronization module is called to initiate the renaming process. During the renaming phase, records from the renaming table are cached. After the renaming phase ends, the cached records are processed first.

[0018] In one embodiment, the data persistence task involves calling the metadata synchronization module to obtain metadata for all datasets in the data source, which is then used to parse data records; converting the message sent by the data ingestion task to the message queue into a format acceptable to the data warehouse module; persisting the logs in the message to the log table of the data warehouse module; and persisting the data to the corresponding data table of the data warehouse module for integration; the log table is in append mode, the data table is in update / insert mode, and each dataset in the data source corresponds to a data table in the data warehouse.

[0019] In one embodiment, the table maintenance task refers to performing maintenance operations on tables within a specified range of multiple data sources in the data warehouse module, deleting data / metadata objects that are no longer needed / referenced to save storage space and accelerate data / metadata access; the table renaming task refers to retrieving all data from the old table and temporary table partitions of the log table in the data warehouse module for the renamed table, determining which data should be deleted from the old table in the data warehouse module and which should be added to the new table, and sending the relevant deleted and added records to the data persistence task for regular persistence processing through a message queue; the table tracing task refers to retrieving all record data from the specified partition of the log table in the data warehouse module according to time for the tracing table, recording the maximum timestamp, and sending these records to the data persistence task for regular persistence processing through a message queue.

[0020] The aforementioned data integration system, as described in this application, integrates all data sources through a source data module, a metadata module, a stream computing module, a data warehouse module, and a batch computing module. The source data module then extracts metadata from the data sources in the source data module and synchronizes, manages, and stores this metadata. The stream computing module then integrates the data sources and metadata, synchronizing the data and metadata from the data sources to the data warehouse module for downstream systems to extract data, thus improving the efficiency of data extraction and usage. Finally, the batch computing module performs table maintenance, table renaming, and table tracing tasks on multiple data sources within the data warehouse module to achieve data persistence, thereby improving the accuracy of data integration within the data warehouse module. Attached Figure Description

[0021] Figure 1 This is a framework diagram of a data integration system in one embodiment;

[0022] Figure 2 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In one embodiment, such as Figure 1 As shown, a data integration system is provided, including a source data module, a metadata module, a stream computing module, a data warehouse module, and a batch computing module;

[0025] The source data module is the source of the integrated data, including relational databases, non-relational databases, and data warehouses; relational databases, non-relational databases, and data warehouses include one or more data sources;

[0026] The metadata module is used to extract metadata from the data source of the source data module, synchronize, manage and store the metadata, and provide a visual management and operation platform;

[0027] The stream computing module is used to integrate data sources and metadata in the form of stream computing, and to synchronize data and metadata from the data source to the data warehouse module;

[0028] The data warehouse module serves as a transit point and destination for data integration, used to store data containing data sources and metadata or to distribute data to downstream systems.

[0029] The batch processing module is used in the data warehouse module to perform table maintenance tasks, table renaming tasks, and table tracing tasks on multiple data sources.

[0030] In the aforementioned data integration system, this application establishes a source data module, a metadata module, a stream computing module, a data warehouse module, and a batch computing module. The source data module integrates all data sources, and the metadata module extracts metadata from the data sources in the source data module, synchronizing, managing, and storing the metadata. The data synchronization service / data ingestion task is responsible for converting this information into a unified format and storing it in the metadata storage. Then, the stream computing module integrates the data sources and metadata, synchronizing the data and metadata from the data sources to the data warehouse module to facilitate data extraction by downstream systems, thus improving the efficiency of data extraction and use. Finally, the batch computing module performs table maintenance, table renaming, and table tracing tasks on multiple data sources in the data warehouse module to achieve data persistence processing, improving the accuracy of data integration in the data warehouse module.

[0031] In one embodiment, the metadata module includes a metadata storage module, a metadata synchronization module, and a metadata management platform. The metadata storage module stores the metadata dataset of the data source. The metadata synchronization module provides multiple data processes for other components to call. The data processes include accessing the data source, data source synchronization, performance processing, table renaming, table tracking, and table maintenance. The metadata management platform synchronizes metadata from the data source to the metadata module and provides functions for managing the metadata in the metadata storage module, including data dashboards, data maps, data lineage, and data tags, for defining data sources and performing data source synchronization.

[0032] In one embodiment, accessing the data source means pulling all dataset information from the data source and synchronously specifying the dataset to be accessed to the metadata storage, creating a data stream and initializing the state information to the data stream state storage module, creating a log table for the data source in the data warehouse module, creating a data table for each dataset according to the metadata, creating a corresponding log / data message queue, and finally creating and starting a pair of data persistence tasks and data ingestion tasks to start data integration.

[0033] Data source synchronization means traversing the specified dataset in the data source to synchronize the metadata in the dataset to the metadata module storage; the metadata includes the dataset's fields, partitions, and constraint information.

[0034] In one embodiment, the performance indicates that new metadata is synchronized and obtained from the data source to create a new version of the data table in the data warehouse module; at the same time, a corresponding log / data message queue is created, the version and table configuration of the data stream are modified in the data stream state storage module to use the new version of the data table, and after the data persistence task finishes consuming the message queue corresponding to the old table, the data persistence task is restarted to start consuming the new table messages from the message queue corresponding to the new table.

[0035] In a specific implementation, the evolution process is designed to adapt to changes in the data source table structure, ensuring that the changed data is correctly retrieved and persisted to the data warehouse. Each time a substantial change occurs to the table structure, a new version table is created in the data warehouse; existing data is converted to records with the new structure and imported into the new version table; then, subsequent data from the data source is consumed in the new format and persisted to the data warehouse. Creating a new version in the data warehouse indicates that a field 'x' has been added to data source table A, triggering the evolution process of this system. The metadata version evolves from n to n+1, requiring the creation of a corresponding v(n+1) version table in the data warehouse.

[0036] In one embodiment, table renaming means waiting for the data ingestion task to complete its last checkpoint, then stopping the task; synchronizing and obtaining new metadata from the data source to create a new version of the data table in the data warehouse module; simultaneously creating a corresponding log / data message queue; waiting for the data persistence task to write all records of the temporary table in the log message queue to the temporary table partition of the data warehouse module, then starting the table renaming task, sending deletion messages to the message queues corresponding to the old table and log table for records that no longer exist in the temporary table, and sending addition messages to the message queues corresponding to the new table and log table for records of the temporary table; waiting for the data persistence task to consume all messages, then stopping the data persistence task, configuring the version and table of the corresponding data stream in the data stream state storage module to use the new version of the data table and resetting the state; finally starting the data persistence task and the data ingestion task to begin consuming the new version of the message queue.

[0037] In a specific embodiment, table renaming is similar to a performance process, both of which handle changes to the source database table structure. The difference is that a performance process directly modifies the source database table structure, and this system can directly receive the change data capture record and trigger the performance process; table renaming involves creating a new table A', deleting the old table A, and then renaming A' to A. For change data capture, it cannot understand the semantics that A' is actually a new version of A. Therefore, this system needs to treat renaming as a performance process to realize the change in table structure and thus achieve persistent data management.

[0038] In one embodiment, table maintenance refers to querying and traversing the database of the data warehouse module from the metadata module, stopping the corresponding data persistence task and data ingestion task and their monitoring, starting the table maintenance task for the data source and waiting for it to complete, and then restoring the corresponding data persistence task and data ingestion task and their monitoring.

[0039] In a specific implementation, if the data warehouse manages historical data in the form of versions, there may be a large number of small files and expired data, resulting in low query and write efficiency and a large amount of disk space consumption; abnormal crashes during system operation may generate orphan files; table maintenance is to solve these problems by deleting unnecessary / illegal metadata / data files and merging and compressing files to improve efficiency and reduce volume.

[0040] In one embodiment, the stream computing module includes a data stream state storage module, a data persistence task, and a data ingestion task; each data source corresponds to a data stream, and the data stream includes a data persistence task and a data ingestion task; the data stream is stored in the data stream state storage module; the stream computing module is used to integrate data from the data source and metadata in the form of stream computing.

[0041] In a specific embodiment, the stream computing module is used to integrate data sources and metadata in the form of stream computing. It mainly starts a data ingestion task to call the metadata synchronization module to obtain the metadata of all datasets of the specified data source from the metadata module to achieve data integration.

[0042] In one embodiment, the data ingestion task involves calling the metadata synchronization module to retrieve metadata for all datasets from a specified data source, which is then used to parse data records. Changes to data and metadata are captured from the specified data source using a change data capture method, and format conversion and data integration are performed. The integrated data records are then sent to the data persistence task for processing via a message queue. The sent data records contain both data and logs, using two separate message queues. When the data stream state changes, the data stream state is synchronized to the data stream state storage. In the performance-driven mode, the latest table structure is parsed from the data records to determine if a performance-driven process has occurred. If so, data stream processing is paused, and the metadata synchronization module is called to initiate the performance-driven process. After the performance-driven process of the metadata synchronization module is completed, the data stream state and table structure are updated before normal record processing resumes. In the table renaming mode, the renaming pattern is identified from the data records, and the metadata synchronization module is called to initiate the renaming process. During the renaming phase, records from the renaming table are cached. After the renaming phase ends, the cached records are processed first.

[0043] In one embodiment, the data persistence task involves calling the metadata synchronization module to obtain metadata for all datasets in the data source, which is then used to parse data records; converting the message sent by the data ingestion task to the message queue into a format acceptable to the data warehouse module; persisting the logs in the message to the log table of the data warehouse module; and persisting the data to the corresponding data table of the data warehouse module for integration; the log table is in append mode, the data table is in update / insert mode, and each dataset in the data source corresponds to a data table in the data warehouse.

[0044] In one embodiment, the table maintenance task refers to performing maintenance operations on tables within a specified range of multiple data sources in the data warehouse module, deleting data / metadata objects that are no longer needed / referenced to save storage space and accelerate data / metadata access; the table renaming task refers to retrieving all data from the old table and temporary table partitions of the log table in the data warehouse module for the renamed table, determining which data should be deleted from the old table in the data warehouse module and which should be added to the new table, and sending the relevant deleted and added records to the data persistence task for regular persistence processing through a message queue; the table tracing task refers to retrieving all record data from the specified partition of the log table in the data warehouse module according to time for the tracing table, recording the maximum timestamp, and sending these records to the data persistence task for regular persistence processing through a message queue.

[0045] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 2As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a data integration system. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0046] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0047] Those skilled in the art will understand that all or part of the processes in the systems described in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0048] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0049] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data integration system, characterized in that, The system includes: a source data module, a metadata module, a stream computing module, a data warehouse module, and a batch computing module; The source data module is the source of the integrated data, including relational databases, non-relational databases, and data warehouses; the relational databases, non-relational databases, and data warehouses include one or more data sources; The metadata module is used to extract metadata from the data source of the source data module, synchronize, manage and store the metadata, and provide a visual management and operation platform; The stream computing module is used to integrate the data source and metadata in the form of stream computing, and to synchronize the data and metadata from the data source to the data warehouse module. The data warehouse module serves as a transit point and destination for data integration, used to store data containing data sources and metadata or to distribute data to downstream systems. The batch computing module is used to perform table maintenance tasks, table renaming tasks, and table tracing tasks on multiple data sources in the data warehouse module. The stream computing module includes a data stream state storage module, a data persistence task, and a data ingestion task; each data source corresponds to a data stream, and the data stream includes a data persistence task and a data ingestion task; the data stream is stored in the data stream state storage module; the stream computing module is used to integrate the data source and metadata in the form of stream computing; The data ingestion task involves calling the metadata synchronization module to retrieve metadata from the metadata module for all datasets of a specified data source, used to parse data records; pulling data and metadata changes from the specified data source using change data capture, performing format conversion and data integration, and then sending the integrated data records to the data persistence task for processing via a message queue; the sent data records contain both data and log parts, using two separate message queues; when the data stream state changes, the data stream state is synchronized to the data stream state storage; in the performance mode, the latest table structure is parsed from the data records to determine if a performance has occurred, and if so, data stream processing is paused and the metadata synchronization module is called to start the performance process; after the metadata synchronization module completes the performance process, the data stream state and table structure are updated before normal record processing resumes; in the table renaming mode, the renaming pattern is identified from the data records, and the metadata synchronization module is called to start the renaming process; during the renaming phase, records from the renaming table are cached; after the renaming phase ends, the cached records are processed first.

2. The system according to claim 1, characterized in that, The metadata module includes a metadata storage module, a metadata synchronization module, and a metadata management platform; the metadata storage module is used to store the metadata of the data source. The metadata synchronization module is used to synchronize metadata from the data source to the metadata module, and provides multiple data processes for other components to call; The data flow includes accessing the data source, data source synchronization, performance entry, table renaming, table tracking, and table maintenance. The metadata management platform is used to provide a visual management function for metadata in the metadata storage module, including data dashboards, data maps, data lineages, and data tags, for defining data sources and performing data source synchronization.

3. The system according to claim 2, characterized in that, The access data source refers to pulling all dataset information from the data source and synchronously specifying the dataset to be accessed to the metadata storage, creating a data stream and initializing the status information to the data stream status storage module, creating a log table for the data source in the data warehouse module, and creating a data table for each dataset according to the metadata, while creating a corresponding log / data message queue, and finally creating and starting a pair of data persistence tasks and data ingestion tasks to start data integration. The data source synchronization refers to traversing the specified dataset in the data source to synchronize the metadata in the dataset to the metadata module storage; the metadata includes the dataset's fields, partitions, and constraint information.

4. The system according to claim 3, characterized in that, The performance process involves synchronizing and obtaining new metadata from the data source, creating a new version of the data table in the data warehouse module, simultaneously creating a corresponding log / data message queue, modifying the data stream version and table configuration in the data stream state storage module to use the new version of the data table, and restarting the data persistence task after it has finished consuming the message queue corresponding to the old table to start consuming the new table messages from the message queue corresponding to the new table.

5. The system according to claim 2, characterized in that, The table renaming indicates that the task will stop after the data ingestion task completes its last checkpoint; new metadata will be synchronized and obtained from the data source to create a new version of the data table in the data warehouse module; and a corresponding log / data message queue will be created at the same time. After the data persistence task writes all records from the temporary table in the log message queue to the temporary table partition of the data warehouse module, the table renaming task is started. Records that no longer exist in the temporary table are sent as deletion messages to the message queues corresponding to the old table and the log table, and temporary table records are sent as addition messages to the message queues corresponding to the new table and the log table. After the data persistence task has consumed all messages, the data persistence task is stopped, and the version and table configuration of the corresponding data stream in the data stream state storage module are configured to use the new version of the data table and the state is reset. The data persistence task and the data ingestion task are started to begin consuming the new version of the message queue.

6. The system according to claim 2, characterized in that, The table maintenance refers to querying and traversing the database of the data warehouse module from the metadata module, stopping the corresponding data persistence tasks and data ingestion tasks and their monitoring, starting the table maintenance task for the data source and waiting for it to complete, and then restoring the corresponding data persistence tasks and data ingestion tasks and their monitoring.

7. The system according to claim 1, characterized in that, The data persistence task involves calling the metadata synchronization module to obtain the metadata of all datasets in the data source for parsing data records; converting the messages sent by the data ingestion task to the message queue into a format acceptable to the data warehouse module; persisting the logs in the data warehouse module to the log table; and persisting the data to the corresponding data table in the data warehouse module for integration. The log table is in append mode, and the data table is in update / insert mode. Each dataset in the data source corresponds to a data table in the data warehouse.

8. The system according to claim 1, characterized in that, The table maintenance task refers to performing maintenance operations on tables within a specified range of multiple data sources in the data warehouse module, deleting data / metadata objects that are no longer needed / referenced, in order to save storage space and accelerate data / metadata access; The table renaming task means retrieving all data from the old table and temporary table partitions of the log table in the data warehouse module for the renamed table, determining which data should be deleted from the old table in the data warehouse module and which should be added to the new table, and sending the relevant deleted and added records to the data persistence task through the message queue for regular persistence processing. The table tracing task refers to retrieving all record data from a specified partition of the log table in the data warehouse module according to time for the table being traced, recording the maximum timestamp, and sending these records to the data persistence task through a message queue for regular persistence processing.

Images