Data governance method, system, device and storage medium for heterogeneous data sources

Through the combination of DataX and Spark/Hive's big data platform, it provides a visual interface and API interface, which solves the problems of cumbersome operations and difficulty in integration in heterogeneous data source governance, and realizes efficient data synchronization and real-time data analysis.

CN118656365BActive Publication Date: 2024-11-22ANHUI CREDIT REPORTING CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411132078.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-22
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

In the prior art, data governance of heterogeneous data sources lacks a visual interface, the operation process is cumbersome, and it is difficult to integrate with other systems, which affects the real-time nature of data analysis.

Method used

Define data acquisition tasks through configuration files, use DataX to provide visual page configuration method and synchronize data, combine Spark and Hive big data platforms for offline governance and real-time storage, generate API interfaces to realize data interaction between multiple systems, and use template method mode and policy mode solution Coupled database configuration.

Benefits of technology

It realizes visual operations of data governance, simplifies the task configuration process, improves the real-time and system integration of data analysis, and reduces task execution time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118656365B_ABST
    Figure CN118656365B_ABST
Patent Text Reader

Abstract

The present application provides a data governance method, system, device and storage medium for heterogeneous data sources, and relates to the field of data analysis technology. The data governance method includes: defining data collection tasks through configuration files; collecting data from a first data source in a heterogeneous data source to obtain a first array; synchronizing the first array to a second data source in a heterogeneous data source based on DataX to obtain a second array; processing the second array based on a big data platform to generate a reusable first target array and generate an API interface. The present application provides a visual interface operation through the heterogeneous data source synchronization tool DataX, and the user only needs to configure parameters on the page without writing the corresponding script code; in the process of generating the first target array, a visual operation interface can be provided through a preset big data platform, and the processing procedure is simple. In the case of a large data scale, the task execution time can be controlled to ensure the real-time nature of data analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and specifically to a data governance method, system, device, and storage medium for heterogeneous data sources. Background Technology

[0002] Heterogeneous data sources refer to multiple data sources with different data structures, access methods, and formats. During the process of information technology construction, due to the phased, technical, and economic factors of various system constructions, users accumulate a large amount of data stored in different ways, ranging from simple file data to complex network databases, thus constituting heterogeneous data sources. In big data processing systems, the data sources in heterogeneous data sources are diverse and the data volume is enormous, requiring efficient and reliable data collection and processing.

[0003] In related technologies, DataX is typically used for data acquisition, and Spark and Hive are used for offline data development. DataX is a general-purpose data acquisition tool that can read from different data sources and write to the target data source. Spark is a fast and general-purpose cluster computing system that is suitable for various data processing scenarios. Hive is based on Hadoop and provides a SQL-like query language for large-scale data analysis.

[0004] These tools support users in data collection, analysis, processing, and mining, providing support for data governance; however, the task definition process of these tools is relatively cumbersome, making data governance of heterogeneous data sources more complex for non-technical personnel and difficult to integrate with other systems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a data governance method, system, device, and storage medium for heterogeneous data sources, solving the problems of lack of visual interfaces and cumbersome operation processes in current data governance.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of this application provide a data governance method for heterogeneous data sources. This data governance method includes: defining data acquisition tasks through configuration files to support the acquisition of structured and unstructured data; acquiring data from a first data source among the heterogeneous data sources based on the data acquisition tasks and a preset acquisition method to obtain a first array; wherein the heterogeneous data sources include MySQL databases, Oracle databases, MongoDB databases, CSV databases, and Hive data warehouses; synchronizing the first array to a second data source among the heterogeneous data sources using a visual page configuration method based on DataX to obtain a second array; performing offline governance, hierarchical warehousing, real-time storage, and job scheduling on the second array based on a preset big data platform to generate a reusable first target array; and configuring and generating API interfaces based on field information in the first target array to complete data interaction between multiple systems according to the API interfaces.

[0008] According to a first aspect of the embodiments of this application, the first data source is one of a MySQL database, an Oracle database, a MongoDB database, and a CSV database, and the second data source is a Hive data warehouse.

[0009] According to a first aspect of the embodiments of this application, the aforementioned configuration of generating an API interface based on field information in a first target array to complete data interaction between multiple systems according to the API interface includes: synchronizing the first target array to a target database based on DataX to obtain a second target array, wherein the query efficiency of the target database is higher than the query efficiency of the Hive data warehouse; processing the second target array through configuration to generate an API interface; wherein the API interface is used to expose the second target array to the outside to complete data interaction between multiple systems.

[0010] According to a first aspect of the embodiments of this application, when the first data source is a MySQL database, the second array is Hive-SQL information; during the offline governance of the second array, the data governance method for heterogeneous data sources further includes: receiving Hive-SQL information through a preset Spark cluster, wherein the Spark cluster is configured to provide a visual operation interface; based on the Hive-SQL information, calling a preset Spark task unit in the Spark cluster; after the Spark task unit receives the Hive-SQL information, executing the Spark task unit through the Spark cluster to generate a corresponding Spark command, the Spark command being used to interact with the corresponding Hive data warehouse.

[0011] According to a first aspect of the embodiments of this application, the big data platform includes an offline analysis unit, a data warehouse unit, a real-time computing unit, and a scheduling unit, and the four are respectively configured to perform offline governance, hierarchical warehouse construction, real-time storage, and job scheduling on the second array based on Spark, Hive, Flink, and xxl-job, so as to generate a reusable first target array.

[0012] According to a first aspect of the embodiments of this application, the aforementioned method of synchronizing a first array to a second data source in a heterogeneous data source using DataX through a visual page configuration to obtain a second array includes: creating a data synchronization page based on the data type and database table information of the heterogeneous data source; wherein the data synchronization page is configured with a first port and a second port, and the two respectively correspond to data source parameters and data destination parameters; after completing the task configuration by setting parameters on the data synchronization page, obtaining data synchronization request information; extracting the data source parameters and data destination parameters from the data synchronization request information, and encapsulating the two to obtain a read plugin and a write plugin; submitting the read plugin and the write plugin to DataX, so that the first array is mapped from the first port to the second port based on DataX, and generating a second array corresponding to the second data source.

[0013] According to a first aspect of the embodiments of this application, in the process of encapsulating data source parameters and data destination parameters, the data governance method for heterogeneous data sources further includes: extracting the common part of the database group configuration and using the common part as a first-level template based on a composite approach combining template method pattern and strategy pattern; wherein, the database group includes a first data source and a second data source; by matching the second-level templates corresponding to each database in the database group with the first-level template, multiple target templates that can differentiate and characterize each database are determined; and each target template is bound to its corresponding preset strategy to achieve system decoupling.

[0014] Secondly, embodiments of this application provide a data governance system for heterogeneous data sources. This system includes a definition module, a collection module, a synchronization module, a processing module, and an interaction module. The definition module defines data collection tasks through configuration files to support the collection of structured and unstructured data. The collection module collects data from a first data source in the heterogeneous data sources based on the data collection tasks and preset collection methods to obtain a first array. The heterogeneous data sources include MySQL databases, Oracle databases, MongoDB databases, CSV databases, and Hive data warehouses. The synchronization module synchronizes the first array to a second data source in the heterogeneous data sources via a visual page configuration method using DataX to obtain a second array. The processing module performs offline governance, hierarchical warehousing, real-time storage, and job scheduling on the second array based on a preset big data platform to generate a reusable first target array. The interaction module configures and generates API interfaces based on field information in the first target array to complete data interaction between multiple systems according to the API interfaces.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the data governance method for heterogeneous data sources described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program or instructions that, when executed by a processor, implement the data governance method for heterogeneous data sources described in the first aspect above.

[0017] This invention provides a data governance method, system, device, and storage medium for heterogeneous data sources. Compared with existing technologies, it has the following advantages:

[0018] This application collects and registers a first array from a first data source in a heterogeneous data source, and then uses the heterogeneous data source synchronization tool DataX to provide a visual interface for operation, synchronizing the first array to a second data source to obtain a second array. DataX supports interactive reading and writing of data in multiple data sources, and users only need to configure parameters on the page, without writing corresponding script code, to complete the data synchronization configuration. In the process of offline data development of the second array to generate a reusable first target array, the preset big data platform can provide a visual operation interface and a simple processing program. When the data scale is large, it can control the task execution time and ensure the real-time performance of data analysis. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a data governance method for heterogeneous data sources provided in an embodiment of this application;

[0021] Figure 2 yes Figure 1 A partial flowchart illustrating the data governance process of the second array in S140;

[0022] Figure 3 yes Figure 1 An exemplary process diagram of S130;

[0023] Figure 4 This is a schematic diagram of the structure of a data governance system for heterogeneous data sources provided in an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an exemplary data governance system for heterogeneous data sources provided in an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0028] This application provides a data governance method, system, device, and storage medium for heterogeneous data sources, which solves the problems of lack of visual interface and cumbersome operation process in current data governance. It realizes visual interface operation, improves the convenience of control tasks, and shortens the execution time to ensure the real-time performance of data analysis.

[0029] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0030] In big data processing systems, the challenge of acquiring data from heterogeneous data sources is frequently encountered. These heterogeneous data sources contain diverse and massive amounts of data, requiring efficient and reliable acquisition and processing. Heterogeneous data sources refer to multiple data sources with different data structures, access methods, and formats. During the process of information technology construction, due to the varying stages, technical aspects, and economic factors involved in system development, users accumulate a large amount of data stored in different ways, ranging from simple file data to complex network databases, thus constituting heterogeneous data sources.

[0031] In related technologies, DataX is typically used for data acquisition to address this challenge, and offline data development is based on Spark and Hive. DataX is an open-source, general-purpose data acquisition tool that can read from and write to target data sources from different data sources. DataX is efficient, flexible, and easily scalable, and can meet complex data acquisition needs. However, DataX lacks a visual interface and requires configuration files to define tasks, making it complex for non-technical personnel. Furthermore, as a standalone tool, DataX is difficult to integrate with other systems; for example, it cannot be directly executed as a scheduled task, which hinders automation and scheduling.

[0032] Besides data acquisition, offline data development is also a crucial step. Common offline data development frameworks include Spark and Hive. Spark is a fast and versatile cluster computing system suitable for various data processing scenarios; Hive, based on Hadoop, provides a SQL-like query language and is used for large-scale data analysis. However, offline data development also faces challenges, such as cumbersome program writing and debugging, and the potential for longer task execution times as data volume increases, affecting the real-time performance of data analysis.

[0033] In summary, during the data collection and governance process of big data processing systems, tools such as DataX, Spark, and Hive support users in data collection, analysis, processing, and mining, providing support for data governance. However, these tools are relatively cumbersome in the task definition process, making data governance of heterogeneous data sources complex for non-technical personnel and difficult to integrate with other systems.

[0034] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0035] The following section first introduces a data governance method for heterogeneous data sources provided in the embodiments of this application.

[0036] This application provides a flowchart illustrating a data governance method for heterogeneous data sources, as shown in the embodiments below. Figure 1 As shown, the data governance method for heterogeneous data sources may include the following steps S110-S150.

[0037] S110. Define data acquisition tasks through configuration files to support the acquisition of structured and unstructured data.

[0038] S120. Based on the data acquisition task and the preset acquisition method, data is acquired from the first data source in the heterogeneous data source to obtain the first array; wherein, the heterogeneous data source includes MySQL database, Oracle database, MongoDB database, CSV database and Hive data warehouse.

[0039] S130. Using a visual page configuration method, the first array is synchronized to the second data source in the heterogeneous data source based on DataX to obtain the second array.

[0040] S140. Based on the preset big data platform, the second array is processed offline, layered warehouse is built, real-time storage is performed, and job scheduling is carried out to generate a reusable first target array.

[0041] S150. Configure and generate API interfaces based on the field information in the first target array, so as to complete data interaction between multiple systems according to the API interfaces.

[0042] In this embodiment, the data processing in heterogeneous data sources includes stages such as data acquisition, data synchronization, data governance, and data services. After data acquisition, a raw first array is obtained. To process the acquired first array uniformly, it can be synchronized to another database in the heterogeneous data source. In this other database, a second array can be processed using a preset data governance tool to obtain a reusable first target array. Then, the field information in the first target array is configured as an API interface. It is understood that an API interface is short for Application Programming Interface, which allows third-party programs to call data or provide data-based services over the network. Through the aforementioned API interface generated based on the first target array, the effective information in the reusable first target array can be exposed to enable interaction between multiple systems.

[0043] It should be noted that DataX, as a heterogeneous data source synchronization tool, provides a visual interface operation. In the process of synchronizing data in the first array, this application does not require writing synchronization task script code and submitting it to DataX on the server via command line for each data synchronization task. This application has a visual page configuration method, making the data synchronization function a page. The data synchronization operation between the first array and the second array based on DataX is also relatively simple for non-technical personnel.

[0044] The above describes the specific implementation of the data governance method for heterogeneous data sources provided in this application embodiment. After collecting and registering the first array from the first data source, the heterogeneous data source synchronization tool DataX provides a visual interface operation and synchronizes the first array to the second data source to obtain the second array. DataX supports data reading and writing interaction in multiple data sources, and users only need to configure parameters on the page without writing corresponding script code to complete the data synchronization configuration. During the offline data development of the second array, the preset big data platform provides a visual operation interface and simple processing procedures. When the data scale is large, the task execution time can be controlled to ensure the real-time performance of data analysis.

[0045] In one example, the big data platform includes an offline analysis unit, a data warehouse unit, a real-time computing unit, and a scheduling unit. These four units are configured to perform offline governance, hierarchical warehouse construction, real-time storage, and job scheduling on the second array based on Spark, Hive, Flink, and xxl-job, respectively, to generate a reusable first target array.

[0046] In some embodiments, the first data source is one of a MySQL database, an Oracle database, a MongoDB database, and a CSV database, and the second data source is a Hive data warehouse.

[0047] In the embodiments of this application, it can be understood that Hive is a data warehouse tool based on Hadoop. The Hive data warehouse can be used to extract, transform, and load data. The Hive data warehouse is a mechanism that can store, query, and analyze large-scale data stored in Hadoop, and can map structured data files to a database table. Based on this, in the process of data governance for heterogeneous data sources, data from databases such as MySQL, Oracle, MongoDB, and CSV can be synchronously mapped to the Hive data warehouse to facilitate offline governance, hierarchical warehouse construction, real-time storage, and job scheduling to generate reusable data information.

[0048] In some embodiments, an API interface is generated based on the field information in the first target array to complete data interaction between multiple systems according to the API interface. That is, S150 may include the following steps:

[0049] S151. Based on DataX, synchronize the first target array to the target database to obtain the second target array. The query efficiency of the target database is higher than that of the Hive data warehouse.

[0050] S152. The second target array is processed through configuration to generate an API interface; wherein, the API interface is used to expose the second target array to the outside world to complete data interaction between multiple systems.

[0051] In the embodiments of this application, it can be understood that the Hive data warehouse is built on Hadoop based on static batch processing. The query operation process of the Hive data warehouse strictly follows the Hadoop MapReduce job execution model. The Hive data warehouse converts the user's HiveQL statement into a MapReduce job through the interpreter and submits it to the Hadoop cluster. Hadoop monitors the job execution process and then returns the job execution result to the user.

[0052] It is important to emphasize that Hadoop typically has high latency and requires significant overhead during job submission and scheduling. Therefore, Hive data warehouses cannot achieve low-latency, fast queries on large-scale datasets.

[0053] Based on this, before configuring the API interface, this application synchronizes the first target array obtained to the target database. The query efficiency of the target database is higher than that of the Hive data warehouse, thereby ensuring the high real-time performance of the job.

[0054] In some embodiments, where the first data source is a MySQL database, the second array contains Hive-SQL information. Please refer to... Figure 2 In the process of offline governance of the second array, this data governance method for heterogeneous data sources may also include the following steps:

[0055] S210: Receive Hive-SQL information through a preset Spark cluster, where the Spark cluster is configured to provide a visual operation interface.

[0056] S220: Based on Hive-SQL information, it calls the preset Spark task unit in the Spark cluster.

[0057] S230. After receiving Hive-SQL information, the Spark task unit executes the Spark task unit through the Spark cluster to generate the corresponding Spark commands. The Spark commands are used to interact with the corresponding Hive data warehouse.

[0058] In this application's embodiments, it is understood that the application integrates the big data processing framework Spark into the system. The system encapsulates Spark task units. After a user submits SQL information on the page, this SQL information is synchronized to the Hive data warehouse via DataX to obtain Hive-SQL information. The system can call the Spark task unit to generate corresponding Spark commands, and interact with the corresponding Hive data warehouse through Spark commands to execute the Hive-SQL information. In this process, the user only needs to write SQL information on the corresponding page of the system, and the system can convert the SQL information into corresponding Spark code and run it. The entire operation process is relatively simple, and even non-professionals can easily operate it.

[0059] In some embodiments, please refer to Figure 3 Through a visual page configuration method, the first array is synchronized to the second data source in the heterogeneous data source based on DataX to obtain the second array. That is, S130 may include the following steps:

[0060] S310. Based on the data types and table information of the heterogeneous data source, create a data synchronization page; wherein, the data synchronization page is configured with a first port and a second port, and the two correspond to the data source parameter and the data destination parameter, respectively.

[0061] S320. After completing the task configuration by setting parameters on the data synchronization page, obtain the data synchronization request information.

[0062] S330. Extract the data source parameters and data destination parameters from the data synchronization request information, and encapsulate them to obtain the read plugin and write plugin.

[0063] S340. Submit the read plugin and write plugin to DataX so that the first array is mapped from the first port to the second port based on DataX, and a second array corresponding to the second data source is generated.

[0064] In this embodiment, the data synchronization function based on DataX is constructed in the form of a page. Users only need to configure the relevant parameters on the data synchronization page to complete the data synchronization task configuration. The data synchronization page realizes the visual data synchronization of heterogeneous data sources. It can be understood that after the user submits the corresponding configuration, the data source parameters and data destination parameters in the request are encapsulated into read plugins and write plugins, and the read plugins and write plugins are stored in the system database. When the data synchronization task needs to be run, the system will submit the read plugins and write plugins to DataX, thereby using DataX to complete the data synchronization task, mapping the first array from the first port to the second port, and generating a second array corresponding to the second data source.

[0065] In some embodiments, the data governance method for heterogeneous data sources may further include the following steps during the encapsulation of data source parameters and data destination parameters:

[0066] S410. Based on a composite approach combining the template method pattern and the strategy pattern, extract the common parts of the database group configuration and use the common parts as a first-level template; wherein, the database group includes a first data source and a second data source.

[0067] S420. By matching the secondary templates corresponding to each database in the database group with the primary templates, multiple target templates that can differentiate and characterize each database are determined.

[0068] S430. Bind each target template to its corresponding preset strategy to achieve system decoupling.

[0069] In the embodiments of this application, it is understood that since heterogeneous data sources include various databases such as MySQL, Oracle, MongoDB, CSV, Hive data warehouse, and DM, the configuration information required for each database is not entirely the same. Based on this, in order to efficiently encapsulate the data source parameters and data destination parameters, this application adopts a combination of template method pattern and strategy pattern. The common part of the database group configuration is extracted and used as a first-level template. Then, the second-level templates corresponding to each database are matched with the first-level template to determine the target template corresponding to each database. It is understood that binding each target template with its corresponding preset strategy can effectively achieve system decoupling. Each target template realizes the differentiated expression of the database.

[0070] This application also provides a data governance system 500 for heterogeneous data sources, such as... Figure 4 As shown, the data governance system 500 may include the following modules:

[0071] Module 510 is defined to define data acquisition tasks via configuration files to support the acquisition of structured and unstructured data.

[0072] The data acquisition module 520 is used to acquire data from the first data source among the heterogeneous data sources based on the data acquisition task and the preset acquisition method, and obtain the first array; wherein, the heterogeneous data sources include MySQL database, Oracle database, MongoDB database, CSV database and Hive data warehouse.

[0073] The synchronization module 530 is used to synchronize the first array to the second data source in the heterogeneous data source through a visual page configuration method based on DataX, so as to obtain the second array.

[0074] The processing module 540 is used to perform offline governance, hierarchical warehousing, real-time storage and job scheduling on the second array based on a preset big data platform, and generate a reusable first target array.

[0075] The interaction module 550 is used to configure and generate API interfaces based on the field information in the first target array, so as to complete data interaction between multiple systems according to the API interfaces.

[0076] According to embodiments of this application, any and multiple modules among the definition module 510, acquisition module 520, synchronization module 530, processing module 540, and interaction module 550 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module.

[0077] In some embodiments, the interaction module 550 may specifically be used for:

[0078] Based on DataX, the first target array is synchronized to the target database to obtain the second target array. The query efficiency of the target database is higher than that of the Hive data warehouse.

[0079] The second target array is processed through configuration to generate an API interface; the API interface is used to expose the second target array to the outside world to complete data interaction between multiple systems.

[0080] In some embodiments, when the first data source is a MySQL database, the second array is Hive-SQL information; the processing module 540 may specifically include:

[0081] The receiving unit 541 is used to receive Hive-SQL information through a preset Spark cluster, wherein the Spark cluster is configured to provide a visual operation interface.

[0082] Calling unit 542 is used to call a pre-defined Spark task unit in the Spark cluster based on Hive-SQL information;

[0083] Command generation unit 543 is used to generate corresponding Spark commands by executing the Spark task unit through the Spark cluster after the Spark task unit receives Hive-SQL information. The Spark commands are used to interact with the corresponding Hive data warehouse.

[0084] In some embodiments, the synchronization module 530 may specifically include:

[0085] Page creation unit 531 is used to create a data synchronization page based on the data type and database table information of heterogeneous data sources; wherein, the data synchronization page is configured with a first port and a second port, which correspond to the data source parameter and the data destination parameter, respectively.

[0086] The request acquisition unit 532 is used to obtain data synchronization request information after the task configuration is completed by setting parameters on the data synchronization page.

[0087] Extraction and encapsulation unit 533 is used to extract the data source parameters and data destination parameters from the data synchronization request information, and encapsulate the two to obtain the read plugin and the write plugin.

[0088] The submission generation unit 534 is used to submit the read plugin and the write plugin to DataX so that the first array is mapped from the first port to the second port based on DataX, and a second array corresponding to the second data source is generated.

[0089] In some embodiments, the extraction and packaging unit 533 can be specifically used for:

[0090] Based on a composite approach combining the template method pattern and the strategy pattern, the common parts of the database group configuration are extracted and used as the first-level template; wherein, the database group includes a first data source and a second data source.

[0091] By matching the secondary templates corresponding to each database in the database group with the primary templates, multiple target templates that can differentiate and characterize each database are determined.

[0092] Each target template is bound to its corresponding preset strategy to achieve system decoupling.

[0093] This application also provides an exemplary data governance system for heterogeneous data sources. Please refer to... Figure 5 Data sources can include MySQL, Oracle, MongoDB, API, CSV, DM, and Hive data warehouses. Data governance begins with data collection from these data sources. Collection methods include offline collection using DataX, real-time collection using Kafka, internet collection using Python, and collection via third-party systems using HTTP / FTP. The big data platform comprises offline computing and analysis units, a data warehouse, a real-time computing unit, and a scheduling unit. The data warehouse includes an ODS layer, a DWD layer, a DWS layer, a DIM layer, a TMP layer, and an APP layer. Raw data collected from the data sources is mapped to the ODS layer of the Hive data warehouse.

[0094] The scheduling unit can perform job scheduling based on Hue / xxl-job, where xxxl-job is a lightweight, high-performance distributed task scheduling framework. xxxl-job is simple to use yet flexible and functional. Through xxxl-job, advanced functions such as scheduled task scheduling, task dependency handling, task failover, and retrying on failure can be implemented, ensuring that each data processing task is executed on time and accurately. In addition, xxxl-job provides real-time monitoring capabilities, including real-time monitoring of task running status and running logs, as well as task alarm functions. If a task fails or an anomaly occurs, the system can promptly notify relevant personnel via email, SMS, etc., greatly improving the efficiency and reliability of task management.

[0095] After performing offline management, hierarchical warehousing, real-time storage, and job scheduling on the aforementioned second array, and generating a reusable first target array, due to the low query efficiency of the Hive data warehouse, in order to ensure the query efficiency of big data, the reusable first target array is exchanged with the target database through DataX. Data is then shared through the target database. The system can provide services such as data interface API, search, and SDK to facilitate data interaction between multiple systems.

[0096] In some embodiments, the aforementioned data governance system for heterogeneous data sources may further include a quality management module, an operation and maintenance module, and a security management module; wherein, the quality management module is used to perform quality monitoring, pattern detection, and data comparison during the data governance process to ensure data quality; the operation and maintenance module is used to perform scheduling monitoring, failure rerun, and error alarms during the data governance process; and the security management module is used to perform access control, operation auditing, and data anonymization during the data governance process.

[0097] This data governance system for heterogeneous data sources is built on the Spring Cloud technology stack. In terms of monitoring, the system integrates a monitoring system and ELK-related technologies. The system uses Prometheus and Grafana for real-time monitoring and performance metric collection and storage, and uses Kibana for log data visualization and retrieval to ensure that the system can promptly alert when there are software or hardware problems, thus ensuring system stability.

[0098] The system also uses ELK for audit log management, ensuring that any problems encountered during system operation are traceable, thereby improving system stability and reliability. In terms of operations and maintenance, the system uses Docker and Kubernetes as its operational framework to manage nodes within the system. Additionally, Jenkins is used as a tool for continuous integration and continuous delivery, enabling automated building, testing, and deployment, thus improving development and deployment efficiency.

[0099] Furthermore, the system uses Docker as a containerization technology to package each microservice into an independent container, achieving environment isolation and deployment consistency, thereby enabling rapid deployment and scaling of services.

[0100] In some embodiments, this application provides an electronic device, the structural schematic of which is shown below. Figure 6 As shown.

[0101] The electronic device may include a processor 610 and a memory 620 storing computer program instructions.

[0102] Specifically, the processor 610 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0103] Memory 620 may include mass storage for data or instructions. For example, and not limitingly, memory 620 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 620 may include removable or non-removable (or fixed) media. Where appropriate, memory 620 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 620 is non-volatile solid-state memory.

[0104] Memory 620 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory 620 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it can perform the operations described in any of the data governance methods for heterogeneous data sources in the above embodiments.

[0105] The processor 610 reads and executes computer program instructions stored in the memory 620 to implement any of the data governance methods for heterogeneous data sources in the above embodiments.

[0106] In one example, the electronic device may also include a communication interface 630 and a bus 600. Wherein, such as Figure 6 As shown, the processor 610, memory 620, and communication interface 630 are connected via bus 600 and communicate with each other.

[0107] The communication interface 630 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0108] Bus 600 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 600 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0109] Furthermore, in conjunction with the data governance methods for heterogeneous data sources described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data governance methods for heterogeneous data sources described in the above embodiments.

[0110] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0111] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0112] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0113] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0114] In summary, compared with the prior art, this application has the following beneficial effects:

[0115] 1. This application provides a visual interface operation through the heterogeneous data source synchronization tool DataX to synchronize the first array to the second data source to obtain the second array; DataX supports data reading and writing interaction in multiple data sources, and users only need to configure parameters on the page, without writing corresponding script code, to complete the data synchronization configuration task.

[0116] 2. In the process of developing offline data for the second array, this application integrates the big data processing framework Spark into the system and provides a visual operation interface. Taking the first data source as a MySQL database and the second array as Hive-SQL information as an example, users only need to write SQL information on the corresponding page of the system, and the system can convert the SQL information into the corresponding Spark code and run it. The whole operation process is relatively simple and can be easily operated by non-professionals.

[0117] 3. Before generating the API interface to complete data interaction between multiple systems, this application synchronizes the first target array to the target database and processes the second target array through configuration to generate the API interface. Since the query efficiency of the target database is higher than that of the Hive data warehouse, the high latency problem of the Hive data warehouse is solved. When the data scale is large, the task execution time can be effectively controlled to ensure the real-time performance of data analysis.

[0118] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data governance method for heterogeneous data sources, characterized in that: include: Define data collection tasks through configuration files to support the collection of structured and unstructured data; Based on the data collection task and the preset collection method, data is collected from a first data source in the heterogeneous data sources to obtain a first array; wherein the heterogeneous data source includes a MySQL database, an Oracle database, a MongoDB database, a CSV database, and a Hive data warehouse; By means of a visual page configuration method, based on DataX, the first array is synchronized to a second data source in a heterogeneous data source to obtain a second array; Based on a preset big data platform, the second array is managed offline, stored in layers, stored in real time, and scheduled, to generate a reusable first target array; Generating an API interface based on the field information configuration in the first target array, so as to complete data interaction among multiple systems according to the API interface; The method of synchronizing the first array to a second data source in a heterogeneous data source based on DataX through a visualized page configuration method to obtain a second array includes: Based on the data type and library table information of the heterogeneous data source, a data synchronization page is created; wherein the data synchronization page is configured with a first port and a second port and the two ports correspond to the data source parameter and the data destination parameter respectively; After completing the task configuration by setting parameters on the data synchronization page, obtaining data synchronization request information; Extracting the data source parameter and the data destination parameter in the data synchronization request information, and encapsulating the two to obtain a read plug-in and a write plug-in; Submitting the read plug-in and the write plug-in to DataX so that the first array is mapped from the first port to the second port based on DataX, and generating a second array corresponding to the second data source; Wherein, in the process of encapsulating the data source parameter and the data destination parameter, the data governance method for heterogeneous data sources further includes: Based on a composite method combining the template method mode and the strategy mode, a common part of the database group configuration is extracted and the common part is used as a primary template; wherein the database group includes the first data source and the second data source; Determine a plurality of target templates that can differentially characterize each database by matching the secondary template corresponding to each database in the database group with the primary template; Each of the target templates is bound to its corresponding preset strategy to achieve system decoupling.

2. The data governance method for heterogeneous data sources according to claim 1, characterized in that: The first data source is one of the MySQL database, the Oracle database, the MongoDB database, and the CSV database, and the second data source is the Hive data warehouse.

3. The data governance method for heterogeneous data sources according to claim 2, characterized in that: The generating an API interface based on the field information configuration in the first target array, so as to complete data interaction among multiple systems according to the API interface, includes: Based on the DataX, the first target array is synchronized to a target database to obtain a second target array, wherein the query efficiency of the target database is higher than the query efficiency of the Hive data warehouse; The second target array is processed by configuration to generate an API interface; wherein the API interface is used to expose the second target array to the outside to complete data interaction between multiple systems.

4. The data governance method for heterogeneous data sources according to claim 3, characterized in that: When the first data source is the MySQL database, the second array is Hive-SQL information; In the process of offline management of the second array, the data management method for heterogeneous data sources further includes: Receiving the Hive-SQL information through a preset Spark cluster, wherein the Spark cluster is configured to provide a visual operation interface; Based on the Hive-SQL information, calling a preset Spark task unit in the Spark cluster; After the Spark task unit receives the Hive-SQL information, the Spark task unit is executed by the Spark cluster to generate a corresponding Spark command, where the Spark command is used to interact with the corresponding Hive data warehouse.

5. The data governance method for heterogeneous data sources according to any one of claims 1 to 4, characterized in that: The big data platform includes an offline analysis unit, a data warehouse unit, a real-time computing unit and a scheduling unit, and the four are respectively configured to perform offline management, hierarchical warehousing, real-time storage and job scheduling on the second array based on Spark, Hive, Flink and xxl-job to generate a reusable first target array.

6. A data governance system for heterogeneous data sources, characterized in that: include: The definition module is used to define data collection tasks through configuration files to support the collection of structured and unstructured data; A collection module, used for collecting data from a first data source in the heterogeneous data sources based on the data collection task and the preset collection method to obtain a first array; wherein the heterogeneous data source includes a MySQL database, an Oracle database, a MongoDB database, a CSV database and a Hive data warehouse; A synchronization module, configured to synchronize the first array to a second data source in a heterogeneous data source based on DataX through a visual page configuration method to obtain a second array; A processing module, used to perform offline management, hierarchical warehouse building, real-time storage and job scheduling on the second array based on a preset big data platform to generate a reusable first target array; An interaction module, configured to generate an API interface based on the field information configuration in the first target array, so as to complete data interaction among multiple systems according to the API interface; The synchronization module is specifically used for: Based on the data type and library table information of the heterogeneous data source, a data synchronization page is created; wherein the data synchronization page is configured with a first port and a second port and the two ports correspond to the data source parameter and the data destination parameter respectively; After completing the task configuration by setting parameters on the data synchronization page, obtaining data synchronization request information; Extracting the data source parameter and the data destination parameter in the data synchronization request information, and encapsulating the two to obtain a read plug-in and a write plug-in; Submitting the read plug-in and the write plug-in to DataX so that the first array is mapped from the first port to the second port based on DataX, and generating a second array corresponding to the second data source; In the process of encapsulating the data source parameters and the data destination parameters, based on a composite method combining the template method pattern and the strategy pattern, the common part of the database group configuration is extracted and the common part is used as a primary template; wherein the database group includes the first data source and the second data source; by matching the secondary template corresponding to each database in the database group with the primary template, a plurality of target templates that can differentially characterize each database are determined; each of the target templates is bound to its corresponding preset strategy to achieve system decoupling.

7. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements a data governance method for heterogeneous data sources as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the data governance method for heterogeneous data sources as described in any one of claims 1 to 5 is implemented.