Distributed cross-domain heterogeneous data exchange system

Abstract

The invention discloses a distributed cross-domain heterogeneous data exchange system. The system comprises a plurality of distributed nodes, each distributed node comprises a data adapter module, a data auditing module, an ETL tool set module, an exchange rule module and a task scheduling module; the data adapter module is used for realizing unified data processing and exchange among heterogeneous data sources; the data auditing module is used for performing data quality inspection on the received data and the processed data to ensure that data acquisition and processing meet quality standards; the ETL tool set module is used for visually designing a data exchange task, executing data extraction and conversion and loading the data exchange task; the exchange rule module is used for defining rules and strategies which need to be followed by the system in the data exchange process; and the task scheduling module is used for managing and executing each task in the data exchange process. The system is a distributed cross-domain heterogeneous data exchange system which is simpler and more efficient, carries out data distributed processing, supports multiple data checking processing mechanisms and guarantees data quality.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a distributed cross-domain heterogeneous data exchange system. Background Art

[0002] The technical background of distributed cross-domain heterogeneous data exchange system is that it emerged in order to achieve efficient, secure and scalable data exchange in the face of different geographical locations, data source types and data format diversity.

[0003] Traditional distributed cross-domain heterogeneous data exchange reflects the complexity and diversity of modern data management and exchange. However, in the face of the complexity and challenges of cross-domain heterogeneous data exchange, traditional data exchange methods usually face the following problems: Data fragmentation: Data is scattered in different geographical locations and data sources, resulting in data fragmentation and difficulty in integration and management. Inconsistent data formats: Different data sources may use different data formats and structures, requiring cumbersome data format conversion. Low transmission efficiency: Data transmission may be limited by network bandwidth, resulting in slow transmission speed. Summary of the invention

[0004] Based on this, it is necessary to provide a distributed cross-domain heterogeneous data exchange system to address the above technical issues.

[0005] A distributed cross-domain heterogeneous data exchange system, the system includes multiple distributed nodes; each distributed node includes a data adapter module, a data audit module, an ETL tool set module, an exchange rule module, and a task scheduling module; the data adapter module is used to realize unified data processing and exchange between heterogeneous data sources; the data audit module is used to perform data quality checks on received data and processed data to ensure that data collection and processing meet quality standards; the ETL tool set module is used to visually design data exchange tasks and execute data extraction, conversion and loading data exchange tasks; the exchange rule module is used to define the rules and policies that the system needs to follow during the data exchange process; the task scheduling module is used to manage and execute various tasks in the data exchange process.

[0006] In one embodiment, the data sources adapted by the data adapter module include database system data, file system data, message system data, Web service and API data, and sensor data.

[0007] In one embodiment, the data adapter modules of each distributed node are connected using a cross-domain data bus.

[0008] In one embodiment, the data quality check on the received data and the processed data includes:

[0009] The ETL tool set module transmits the processed and cleaned data to the data quality audit module;

[0010] The data quality audit module performs a data quality check on the received data;

[0011] The data quality audit module generates a data quality report, which includes the results of the data quality check, including the passed and failed check items, and related statistical information;

[0012] The data quality report is passed to other systems or users for further decision making and data processing.

[0013] In one embodiment, the ETL toolset module provides an ETL toolset and supports online visual ETL process design; the common ETL toolset includes a string processing tool, a replacement tool, a null judgment tool, a calculator tool, a merge tool, a grouping tool, a sorting tool, and a record generation tool.

[0014] In one embodiment, the exchange rule module provides an exchange policy rule setting function, which can set policies for each operation in the data exchange process; all policies of the exchange rule module are stored in a platform policy library for subsequent reuse.

[0015] In one embodiment, the task scheduling module provides unified scheduling management of data acquisition tasks and data exchange tasks, and supports real-time, periodic, regular, script, and file-triggered scheduling of tasks.

[0016] In one embodiment, the system is also used to perform standardized transformation on various types of heterogeneous data, including:

[0017] The data adapter module collects the heterogeneous data from various data sources and transmits it to the ETL tool set module;

[0018] The ETL tool set module extracts, transforms and loads the heterogeneous data to obtain the data to be standardized;

[0019] The ETL tool set module performs normalization processing on the data to be standardized to generate internal standardized data for data exchange;

[0020] The internal standardized data is classified and stored according to dimensions and fact data to form subject data for use.

[0021] In one embodiment, the unified scheduling management of the acquisition task and the data exchange task includes:

[0022] The task scanner selects tasks and puts them into the task queue;

[0023] The computing node monitors the task queue and puts the task into the task scheduler;

[0024] The task scheduler receives the task and assigns the task to a task thread pool;

[0025] The task thread in the task thread pool executes the task, and after the execution is completed, the task thread is returned to the task thread pool;

[0026] The resource listener monitors the resource queue and puts the resources into the resource scheduler;

[0027] The resource scheduler receives the resources and allocates them to a resource thread pool;

[0028] The resource threads in the resource thread pool execute the resources;

[0029] If the resource status is completed, the resource will be executed and then the resource thread will be returned to the resource thread pool;

[0030] If the resource status is not completed, the resource will be placed in a blocking queue to wait for subsequent processing.

[0031] In one embodiment, the system is further used for data quality processing, including:

[0032] The various data sources provide the heterogeneous data;

[0033] The data adapter module accesses the heterogeneous data and transmits it to the data audit module for collecting data inspection;

[0034] After the data audit module returns the collection audit result, the heterogeneous data is transferred to the ETL tool set module;

[0035] The ETL tool set module performs standardization processing on the heterogeneous data, converts the heterogeneous data into a standard format, and obtains the internal standardized data;

[0036] The ETL toolset module transfers the internal standardized data to the data audit module to perform data quality check;

[0037] The data audit module transmits high-quality data that has passed the data quality audit to the target system for further processing and analysis.

[0038] The system provided by the present invention realizes data integration, and can integrate data into a unified platform across different geographical locations and data sources to facilitate data management and analysis; realizes data format conversion and mapping, so that the system can automatically perform data format conversion and mapping to ensure data consistency and availability; improves transmission efficiency, especially in cross-domain situations, optimizes data transmission and synchronization processes, and improves the efficiency and speed of data exchange; solves the complexity in distributed, cross-domain and heterogeneous data exchange to achieve more efficient, safer and more controllable data exchange and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A distributed cross-domain heterogeneous data exchange system structure is provided in one embodiment;

[0040] Figure 2 A diagram of a cross-domain data exchange transmission structure provided in one embodiment;

[0041] Figure 3 A diagram of a heterogeneous data processing process provided in one embodiment;

[0042] Figure 4 A data quality assurance process diagram provided in one embodiment;

[0043] Figure 5 A diagram of a data audit process provided in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, please refer to Figure 1 , provides a distributed cross-domain heterogeneous data exchange system, the system includes multiple distributed nodes; each distributed node includes a data adapter module, a data audit module, an ETL tool set module, an exchange rule module, and a task scheduling module; the data adapter module is used to realize unified data processing and exchange between heterogeneous data sources; the data audit module is used to perform data quality checks on received data and processed data to ensure that data collection and processing meet quality standards; the ETL tool set module is used to visually design data exchange tasks and execute data extraction, conversion and loading data exchange tasks; the exchange rule module is used to define the rules and policies that the system needs to follow during the data exchange process; the task scheduling module is used to manage and execute various tasks in the data exchange process.

[0046] Specifically, the distributed cross-domain heterogeneous data exchange system adopts a symmetrical design. The system supports the collection and access of various heterogeneous data sources through data adapters. After access, the data is immediately audited to verify whether the collected data meets the standards.

[0047] The ETL function module is used to pre-process the data to generate the intermediate standard value of the data, so that the subsequent data distributed scheduling and transmission operations are smoother. According to the data exchange rules and strategies set by the user, the system forms a data workflow and transmits the data to the target end.

[0048] If customized data processing is required on the target side, users can perform secondary data processing when the data exchange workflow reaches the ETL function module. When all data processing is completed, the data quality audit function will be called again to check whether there are any problems such as data loss, data duplication, and data shift in the reconciliation processing data to ensure that the data processing meets the requirements of the target side. The above process is designed to provide a more flexible and customizable data processing method to meet various data processing needs and ensure that the data processing process is more efficient and accurate.

[0049] The target end refers to the final destination or recipient of the data being transmitted, exchanged, loaded or imported. This target end can be a variety of different data storage, processing or application systems, depending on the purpose and requirements of the data exchange.

[0050] Database systems: Data can be exchanged to relational databases, NoSQL databases, or data warehouses for subsequent analysis, querying, and reporting.

[0051] Data warehouse: A data storage system specifically designed for data analysis and reporting, used to support business intelligence and data warehouse analytics.

[0052] Analysis tools: Data can be imported into various analysis tools, such as data visualization tools, statistical analysis software, machine learning platforms, etc., for in-depth data analysis and mining.

[0053] Applications: Data can be transferred to different applications to support application functionality and business logic.

[0054] Cloud services: Data can be uploaded to cloud storage or cloud computing services, such as cloud databases, cloud storage, cloud analysis services, etc.

[0055] Message queues and message buses: used for real-time data transmission and asynchronous communication. Data can be sent to a message queue or message bus for other systems to subscribe to and consume.

[0056] File system: Data can be stored as files for later processing, backup or archiving.

[0057] Data Lake: A data lake is a repository for raw, unprocessed data that can be transferred to the data lake for further processing and analysis.

[0058] Real-time stream processing systems: Data can be sent to real-time stream processing systems for real-time processing and analysis.

[0059] In one embodiment, the data sources adapted by the data adapter module include database system data, file system data, message system data, Wed service and API data, and sensor data. The data adapter: The adapter is a concept introduced to solve the problem of accessing data sources of various types and storage forms. The significance is that different data resources can be connected to the exchange platform. By sorting and classifying the data sources, corresponding adapters are provided for data acquisition / writing for each category of data source, and the content of data exchange is converted into a unified intermediate data format, so as to realize unified data processing and exchange between heterogeneous data sources.

[0060] Specifically, database system data includes relational databases (such as MySQL, Oracle, SQL Server), NoSQL databases (such as MongoDB, Redis), and various other database types. File system data includes: various file formats, such as text files, CSV files, Excel files, JSON files, XML files, log files, etc. Message system data includes: message queues, message buses, used to achieve real-time data transmission and asynchronous communication, such as Apache Kafka, etc. Web service and API data includes: Web services and APIs accessed through HTTP / HTTPS protocols, such as RESTfulAPI, SOAP services, etc. Sensor data includes: data from various sensor devices, such as Internet of Things (IoT) sensors, sensor arrays, etc.

[0061] In one embodiment, please refer to Figure 2 , the data adapter modules of each distributed node are connected by a cross-domain data bus. Cross-domain data exchange and transmission can achieve efficient and secure data transmission between different network domains through a distributed framework. The key to cross-domain data exchange and transmission is to overcome network boundaries and obstacles to ensure that data can flow freely in various network fields. The implementation of cross-domain data exchange and transmission involves multiple key steps. First, through data adapters, we can adapt and parse data from various heterogeneous data sources to ensure that the data can be correctly understood and processed by the system. Next, using ETL (Extract, Transform, Load) tools, we standardize the data so that it has a consistent data structure and format, which helps with subsequent data fusion and processing.

[0062] The uniqueness of cross-domain data exchange and transmission lies in its distributed nature. It not only allows data to flow between different network domains, but also provides data preprocessing and standardization through data adaptation and ETL tools, thus providing a better foundation for subsequent data processing. Therefore, cross-domain data exchange and transmission has broad application prospects in achieving data integration, sharing and collaboration.

[0063] In one embodiment, the data quality check on the received data and the processed data includes:

[0064] The ETL tool set module transmits the processed and cleaned data to the data quality audit module;

[0065] The data quality audit module performs a data quality check on the received data;

[0066] The data quality audit module generates a data quality report, which includes the results of the data quality check, including the passed and failed check items, and related statistical information;

[0067] The data quality report is passed to other systems or users for further decision making and data processing.

[0068] In one embodiment, the ETL toolset module provides an ETL toolset and supports online visual ETL process design; the common ETL toolset includes a string processing tool, a replacement tool, a null judgment tool, a calculator tool, a merge tool, a grouping tool, a sorting tool, and a record generation tool.

[0069] Specifically, ETL toolset: ETL toolset is a tool suite for data standardization and normalization, which is used for data normalization after access through the adapter. Through adapter and ETL visual design, data processing, standardized processing of heterogeneous databases, data files and messages, data integration and sharing are achieved. The platform provides a variety of common ETL toolsets by default.

[0070] Specifically, online visual ETL process design: In order to quickly establish data exchange process jobs, the system supports online visual ETL process design, including: node dragging and dropping, mapping connection design, node-based ETL processing, visual job error checking, support for exchange job saving, support for template creation, support for template application, support for job conversion to template, support for visual interface viewing of scheduling logs, support for real-time job flow status, and support for job error process monitoring

[0071] Specifically, the embedded code editor not only supports the graphical design interface of ETL, but also provides an embedded online J code editor to facilitate advanced users to edit codes online and flexibly customize data exchange services, including: Java code editing, Python code editing, and Shell code editing.

[0072] In one embodiment, the exchange rule module provides an exchange policy rule setting function, which can set policies for each operation in the data exchange process; all policies of the exchange rule module are stored in a platform policy library for subsequent reuse.

[0073] In one embodiment, the task scheduling module provides unified scheduling management of data acquisition tasks and data exchange tasks, and supports real-time, periodic, regular, script, and file-triggered scheduling of tasks.

[0074] In one embodiment, please refer to Figure 3 The system is also used to perform standardized transformation on various types of heterogeneous data, including:

[0075] The data adapter module collects the heterogeneous data from various data sources and transmits it to the ETL tool set module;

[0076] The ETL tool set module extracts, transforms and loads the heterogeneous data to obtain the data to be standardized;

[0077] The ETL tool set module performs normalization processing on the data to be standardized to generate internal standardized data for data exchange;

[0078] The internal standardized data is classified and stored according to dimensions and fact data to form subject data for use.

[0079] Specifically, for various types of heterogeneous data, data collection, data processing, data standardization and subject storage are used to realize the standardized transformation of heterogeneous data, which is convenient for subsequent business integration and analysis. Among them, data collection: mainly through data adapters to parse various data sources, including database systems, file systems and message systems, and introduce data collection into the data exchange system. Among them, data adapters can identify and understand various types of data, including structured and unstructured data, text, images, audio, etc. These components can convert data from its original format into a unified internal data model. Data processing: for the collected data, through the ETL function module, the collected data is extracted, transformed, loaded and processed so that the collected data meets the requirements of the next step of standardization. Data standardization: mainly through the penetration of data standards and rules, the data is standardized and processed to generate internal standardized data for data exchange. According to this method, the system allows data from different sources to be integrated and merged to create a comprehensive data set. This helps to discover the associations and trends between data and provide deeper insights. Subject storage: the standardized data is classified and stored according to dimensions and fact data to form subject data, which is closer to the business and accelerates the subsequent business integration analysis.

[0080] The above data processing process not only improves the quality and consistency of data, but also promotes the integration, standardization and thematic storage of data, thus providing a more beneficial data foundation for business analysis and decision-making. This process helps to improve data management efficiency, accelerate business processes, and bring deeper insights and competitive advantages to organizations.

[0081] In one embodiment, the unified scheduling management of the acquisition task and the data exchange task includes:

[0082] The task scanner selects tasks and puts them into the task queue; specifically, the task scanner periodically scans the database task model library, selects tasks that need to be executed, and puts them into the task queue to wait for execution.

[0083] The computing node listens to the task queue and puts the task into the task scheduler; specifically, computing node 1 and other computing nodes (such as computing node 2 to computing node N) listen to the task queue to see if there are executable tasks. Once an executable task is found, the computing node obtains the task and submits it to the task scheduler to ensure that the task is coordinated and allocated in a distributed environment.

[0084] The task scheduler receives the task and assigns the task to the task thread pool. Specifically, after receiving the new task, the task scheduler is responsible for assigning the task to the task thread pool. The idle threads in the task thread pool will be used to execute the task to improve the concurrent execution efficiency of the task.

[0085] The task thread in the task thread pool executes the task, and after the execution is completed, the task thread is returned to the task thread pool; specifically, the threads in the task thread pool execute tasks, including initialization of tasks, placing resources into queues, execution of tasks, feedback of task execution results to clients, etc. After the task is executed, the task thread is returned to the task thread pool to execute subsequent tasks.

[0086] The resource listener listens to the resource queue and puts the resources into the resource scheduler; specifically, computing node 2 and other computing nodes (such as computing node 3 to computing node N) listen to the resource queue to detect whether there are available resources. If there are available resources, the computing node acquires the resources and submits them to the resource scheduler to coordinate the allocation and execution of resources.

[0087] The resource scheduler receives the resource and allocates it to the resource thread pool; specifically, after receiving the resource, the resource scheduler is responsible for allocating the resource to the idle resource thread in the resource thread pool. The threads in the resource thread pool will perform resource operations to meet the resource processing required by the task.

[0088] The resource threads in the resource thread pool execute the resources;

[0089] If the resource status is completed, the resource will be executed and then the resource thread will be returned to the resource thread pool;

[0090] If the resource status is not completed, the resource will be placed in a blocking queue to wait for subsequent processing.

[0091] In one embodiment, please refer to Figure 4 The system is also used for data quality processing, including:

[0092] The various data sources provide the heterogeneous data;

[0093] The data adapter module accesses the heterogeneous data and transmits it to the data audit module for collecting data inspection;

[0094] After the data audit module returns the collection audit result, the heterogeneous data is transferred to the ETL tool set module;

[0095] The ETL tool set module performs standardization processing on the heterogeneous data, converts the heterogeneous data into a standard format, and obtains the internal standardized data;

[0096] The ETL toolset module transfers the internal standardized data to the data audit module to perform data quality check;

[0097] The data audit module transmits high-quality data that has passed the data quality audit to the target system for further processing and analysis.

[0098] The above data quality processing ensures that data maintains high quality during collection, processing and transmission, which helps to improve the reliability and accuracy of decision-making and data analysis.

[0099] In one embodiment, please refer to Figure 5 , provides a data audit process diagram, data quality audit: used to monitor and evaluate the quality of exchanged data. It can check the accuracy, completeness, consistency and availability of data, and generate quality reports for reference, including: triggering alarms, automated repair processes, and data quality reports.

[0100] Specifically, the data standardization module passes the processed and cleaned data to the data quality audit module, and these data have been converted into a standard format. The data quality audit module performs data quality checks on the received data. These checks may include the following: Data integrity: Check whether the data has missing values ​​or incomplete records. Data accuracy: Verify the accuracy of the data and check whether the value is within a reasonable range. Data consistency: Check whether the data is consistent with known rules and reference data. Data uniqueness: Ensure that there are no duplicate records in the data. Data format: Verify whether the data conforms to the specified format and data type. The data quality audit module generates a data quality report, which includes the results of the data quality check, including passed and failed check items, and related statistical information. The data quality report can be passed to other systems or users for further decision making and data processing. Reports can help organizations identify and solve data quality issues and ensure the credibility and availability of data.

[0101] The system provided by the present invention realizes data integration, and can integrate data into a unified platform across different geographical locations and data sources to facilitate data management and analysis; realizes data format conversion and mapping, so that the system can automatically perform data format conversion and mapping to ensure data consistency and availability; improves transmission efficiency, especially in cross-domain situations, optimizes data transmission and synchronization processes, and improves the efficiency and speed of data exchange; solves the complexity in distributed, cross-domain and heterogeneous data exchange to achieve more efficient, safer and more controllable data exchange and management.

[0102] The present invention realizes efficient data collection: through the intelligent recognition and parsing capabilities of the data adapter, data collection becomes efficient and accurate. This enables the system to easily collect data from various data sources, including structured and unstructured data, multimedia data, etc., thereby improving the comprehensiveness and accuracy of data collection. Data quality assurance: The ETL function module in the data processing process ensures that the collected data undergoes a series of extraction, transformation and loading steps to meet the requirements of standardization. This helps to improve the quality, consistency and reliability of the data and reduce the risk of errors and data problems. Standardized data model: During the data standardization stage, the data is processed according to internal rules and standards to generate an internal standardized data model. This model helps to achieve the integration and fusion of data from different sources, creates a consistent data structure, and makes data analysis and insights easier and more feasible. Comprehensive data insight: Through thematic storage, data is classified and stored according to dimensions and factual data, which helps to form a thematic data set. This operation can better meet business needs and provide data on specific topics to support subsequent business integration and analysis. This accelerates the generation of data insights and helps businesses make decisions faster.

[0103] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A distributed cross-domain heterogeneous data exchange system, characterized in that: The system includes multiple distributed nodes; each distributed node includes a data adapter module, a data audit module, an ETL tool set module, an exchange rule module, and a task scheduling module; The data adapter module is used to realize unified data processing and exchange between heterogeneous data sources; the data audit module is used to perform data quality checks on received and processed data to ensure that data collection and processing meet quality standards; the ETL tool set module is used to visually design data exchange tasks and perform data extraction, conversion and loading data exchange tasks; the exchange rule module is used to define the rules and policies that the system needs to follow during the data exchange process; The task scheduling module is used to manage and execute various tasks in the data exchange process.

2. The system according to claim 1, characterized in that The data sources adapted by the data adapter module include database system data, file system data, message system data, Web service and API data, and sensor data.

3. The system according to claim 2, characterized in that The data adapter modules of each distributed node are connected by a cross-domain data bus.

4. The system according to claim 3, characterized in that The data quality check of the received data and the processed data includes: The ETL tool set module transmits the processed and cleaned data to the data quality audit module; The data quality audit module performs a data quality check on the received data; The data quality audit module generates a data quality report, which includes the results of the data quality check, including the passed and failed check items, and related statistical information; The data quality report is passed to other systems or users for further decision making and data processing.

5. The system according to claim 4, characterized in that The ETL toolset module provides an ETL toolset and supports online visual ETL process design; the ETL toolset includes a string processing tool, a replacement tool, a null judgment tool, a calculator tool, a merging tool, a grouping tool, a sorting tool, and a record generation tool.

6. The system according to claim 5, characterized in that The exchange rule module provides an exchange policy rule setting function, and performs policy setting for each operation in the data exchange process; all policies of the exchange rule module are stored in the platform policy library for subsequent reuse.

7. The system according to claim 6, characterized in that The task scheduling module provides unified scheduling management of data acquisition tasks and data exchange tasks, and supports real-time, periodic, regular, script, and file-triggered scheduling of tasks.

8. The system according to claim 7, characterized in that The system is also used to perform standardized transformation on various types of heterogeneous data, including: The data adapter module collects the heterogeneous data from various data sources and transmits it to the ETL tool set module; The ETL tool set module extracts, transforms and loads the heterogeneous data to obtain the data to be standardized; The ETL tool set module performs normalization processing on the data to be standardized to generate internal standardized data for data exchange; The internal standardized data is classified and stored according to dimensions and fact data to form subject data for use.

9. The system according to claim 8, characterized in that The unified scheduling management of the acquisition tasks and data exchange tasks includes: The task scanner selects tasks and puts them into the task queue; The computing node monitors the task queue and puts the task into the task scheduler; The task scheduler receives the task and assigns the task to a task thread pool; The task thread in the task thread pool executes the task, and after the execution is completed, the task thread is returned to the task thread pool; The resource listener monitors the resource queue and puts the resources into the resource scheduler; The resource scheduler receives the resources and allocates them to a resource thread pool; The resource threads in the resource thread pool execute the resources; If the resource status is completed, the resource will be executed and then the resource thread will be returned to the resource thread pool; If the resource status is not completed, the resource will be placed in a blocking queue to wait for subsequent processing.

10. The system according to claim 9, characterized in that The system is also used for data quality processing, including: The various data sources provide the heterogeneous data; The data adapter module accesses the heterogeneous data and transmits it to the data audit module for collecting data inspection; After the data audit module returns the collection audit result, the heterogeneous data is transferred to the ETL tool set module; The ETL tool set module performs standardization processing on the heterogeneous data, converts the heterogeneous data into a standard format, and obtains the internal standardized data; The ETL toolset module transfers the internal standardized data to the data audit module to perform data quality check; The data audit module transmits high-quality data that has passed the data quality audit to the target system for further processing and analysis.

Images