A multi-source heterogeneous data integration method and system for data integration middleware

Abstract

The present invention relates to the field of data processing, and specifically to a multi-source heterogeneous data integration method and system for data integration middleware. A multi-source heterogeneous data integration system for data integration middleware comprises: a data extraction module, a data processing module for data to be integrated, and a data storage module. The present invention introduces a credibility evaluation mechanism for data sources, calculates error rates, completeness rates, and combines update frequency and historical quality scores to generate a comprehensive credibility score for each data source; this score is used for weighted processing of overlapping data sets from different data sources, ensuring accuracy and credibility of data during the data integration process, effectively solving the problem of data inconsistency caused by different data source quality in traditional data integration technology, and improving data reliability.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multi-source heterogeneous data integration method and system of a data integration middleware. Background Art

[0002] In today's information age, data integration and management have become important challenges facing enterprises and organizations. With the diversification of business systems and the heterogeneity of data sources, how to effectively integrate data from different sources and formats has become the key to improving data utilization efficiency and decision-making quality. Existing data integration technologies usually face problems such as incomplete data extraction and difficulty in ensuring data consistency when processing multi-source heterogeneous data.

[0003] Traditional data integration methods often rely on fixed data extraction rules and preset data conversion logic, which are not flexible and efficient enough when facing frequent updates of data sources and changes in data structures. At the same time, in terms of data storage, traditional data warehouse technology may not be able to meet the storage and rapid access requirements of large-scale data sets, especially when the amount of data is growing rapidly.

[0004] In this context, the present invention proposes a multi-source heterogeneous data integration method for data integration middleware, aiming to improve the efficiency and quality of data integration through innovative technical means and meet the needs of modern data management. Summary of the invention

[0005] The present invention introduces a credibility assessment mechanism for data sources, calculates error rate and completeness rate, and combines update frequency and historical quality scores to generate a comprehensive credibility score for each data source. This score is used for weighted processing of overlapping data sets from different data sources, ensuring accuracy and credibility of data during data integration, effectively solving the problem of data inconsistency caused by different quality of data sources in traditional data integration technology, and improving data reliability.

[0006] A multi-source heterogeneous data integration method of a data integration middleware, comprising:

[0007] S1: Determine the data source to be integrated, identify the type of the data source to be integrated, and perform corresponding connection operations on each data source to be integrated according to the type of the data source to be integrated; based on the data volume and update frequency of the data source to be integrated, perform a full data extraction operation for the data source to be integrated whose data volume is less than the data volume threshold and whose update frequency is less than the frequency threshold; perform an incremental extraction operation for the remaining data sources to be integrated; store all extracted data in an intermediate storage area; clean the data in the intermediate storage area, and then convert the cleaned data into a unified format to obtain the data to be integrated;

[0008] S2: Perform credibility evaluation on each data source to be integrated to obtain a credibility score for each data source to be integrated; for overlapping data sets from different data sources to be integrated in the data to be integrated, perform weighted processing on the overlapping data sets according to the credibility scores of each data source to be integrated to obtain weighted combined data; use the weighted combined data to replace the overlapping data sets in the data to be integrated to obtain complete data;

[0009] S3: Define the target data model, and perform standardization operations on the complete data according to the target data model to obtain standardized data; integrate the standardized data to obtain a unified data set, and store the unified data set in the data warehouse.

[0010] Preferably, the specific operation of performing credibility assessment on each data source to be integrated is:

[0011] For any data source to be integrated, extract sample data from the data source to be integrated; compare the sample data with known reliable data to calculate the error rate; calculate the ratio of missing values ​​and null values ​​in the sample data to calculate the completeness rate; calculate the average of the error rate and the completeness rate, and then multiply the average by 100 to obtain the quality score of the data source to be integrated; based on the update frequency of the data source to be integrated, obtain the update score of the data source to be integrated, the higher the update frequency, the higher the update score; compare the historical data of the data source to be integrated with the known accurate historical data to obtain the historical quality score of the data source to be integrated;

[0012] For any data source to be integrated, use the formula Calculate and obtain the credibility score R of the data source to be integrated, where A, B and c are the quality score, update score and historical quality score of the data source to be integrated respectively; , and They are the weights of quality score, update score and historical quality score respectively.

[0013] Preferably, the specific operation of weighted processing of overlapping data is as follows according to the comprehensive scoring results of each data source to be integrated:

[0014] For any coincident data set x i , i=1, 2, ..., n; where n indicates that the source of the current overlapping data set is n data sources to be integrated; using the formula Calculate and obtain the coincident data set x i The weighted sum of data X; where R i Represents the credibility score of the i-th data source to be integrated.

[0015] Preferably, the specific operations for standardizing the complete data include: determining the data type and format specification of the target field; creating a mapping relationship document for each data source to be integrated, recording the correspondence between the fields of the data source to be integrated and the target fields; determining the target field corresponding to each field of the data source to be integrated; using a data conversion function library to convert the fields in the complete data into the data types defined in the target data model according to the mapping relationship and the target data model; and performing structural conversion on the complete data according to the target data model to obtain standardized data.

[0016] Preferably, all operations on the integrated data sources are stored via blockchain technology.

[0017] Preferably, the specific operation of storing all operations of the integrated data source through blockchain technology is:

[0018] Select a blockchain platform to record and store data operation logs on the blockchain. The data operation logs include: creation, extraction, update, and deletion operations of data in the data source to be integrated, as well as the specific time of the operation, the identity of the user who performed the operation, and the specific content of the data change;

[0019] Generate a hash value for each data operation log, and store the generated hash value on the blockchain through a smart contract;

[0020] Set a fixed time interval. At each time interval, read the stored data operation log hash value from the blockchain and compare it with the locally generated hash value. If the two are consistent, it means that the data operation log has not been tampered with. If they are inconsistent, it means that the data operation log may have been tampered with or is incorrect, and response measures need to be taken.

[0021] Preferably, part of the data from the data source to be integrated can be stored through a distributed cache system, specifically:

[0022] Select a distributed cache system; traverse all data sources to be integrated, obtain historical access records of the data sources to be integrated, and calculate the access frequency of the data sources to be integrated; set a high-frequency threshold, and classify the data sources to be integrated whose access frequency is greater than the high-frequency threshold as high-frequency access data sources; set the data expiration time according to the update frequency of the high-frequency access data sources;

[0023] When there is a data read request, the target data is first read from the distributed cache system; if the target data exists in the distributed cache system, the cached target data is directly returned; if the target data does not exist in the distributed cache system, the target data is read from the data warehouse, and it is determined whether the source of the target data is a high-frequency access data source. If so, the read target data is stored in the distributed cache system at the same time; if not, no operation is performed;

[0024] When the cache time of cached data in the distributed cache system exceeds the data expiration time, the cached data is deleted.

[0025] Preferably, a multi-source heterogeneous data integration system of a data integration middleware comprises:

[0026] The data extraction module is used to identify the type of data source to be integrated, and perform corresponding connection operations on each data source to be integrated according to the type of data source to be integrated; based on the data volume and update frequency of the data source to be integrated, perform full or incremental extraction operations on the data source to be integrated; store all extracted data in the intermediate storage area; clean the data in the intermediate storage area, and then convert the cleaned data into a unified format to obtain the data to be integrated;

[0027] The data processing module to be integrated is used to evaluate the credibility of each data source to be integrated, and to perform weighted processing on the overlapping data sets from different data sources to be integrated according to the credibility score of each data source to be integrated to obtain weighted combined data; and to use the weighted combined data to replace the overlapping data sets in the data to be integrated to obtain complete data;

[0028] The data storage module is used to perform standardization operations on the complete data according to the target data model to obtain standardized data; and to integrate the standardized data and store it in the data warehouse.

[0029] The present invention has the following advantages:

[0030] 1. The present invention introduces a credibility assessment mechanism for data sources, calculates error rate and completeness rate, and combines update frequency and historical quality scores to generate a comprehensive credibility score for each data source. This score is used to weightedly process overlapping data sets from different data sources, ensuring accuracy and credibility of data during data integration, effectively solving the problem of data inconsistency caused by different quality of data sources in traditional data integration technology, and improving data reliability.

[0031] 2. The present invention optimizes data storage and access by adopting blockchain technology and distributed cache system. Through blockchain technology, the transparency and non-tamperability of data operations can be ensured, and the security and traceability of data can be enhanced. At the same time, the distributed cache system can significantly improve the data reading speed for high-frequency access data sources, reduce the need for direct access to the data warehouse, thereby reducing the burden on the data warehouse and improving the response speed and performance of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the structure of a multi-source heterogeneous data integration system of the data integration middleware adopted in an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable persons skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0034] Embodiment 1, a multi-source heterogeneous data integration method of a data integration middleware, comprising:

[0035] S1: Determine the data source to be integrated, identify the type of the data source to be integrated, and perform corresponding connection operations on each data source to be integrated according to the type of the data source to be integrated, for example: for a relational database, use the database connection string to connect and configure the necessary authentication information; for a NoSQL database, use the corresponding connection driver and configuration file to connect; for a file system, configure the configuration file path and access permissions; for an API interface, configure the API endpoint and authentication information; for a real-time data stream, configure the stream processing framework and consumer group; based on the data volume and update frequency of the data source to be integrated, for the data source to be integrated whose data volume is less than the data volume threshold and whose update frequency is less than the frequency threshold, perform a full extraction operation, that is, extract all data each time; for the remaining data sources to be integrated, perform an incremental extraction operation, extracting only the data since the last time Extract new or updated data after extraction; Use SQL query statements to extract data from relational databases, and use ETL tools to perform extraction tasks; Use corresponding query languages ​​or APIs to extract data from NoSQL databases; Read file contents and parse data formats for file systems; Send HTTP requests to obtain data and parse responses for API interfaces; Set up data consumers for real-time data streams to receive and process data streams in real time; Store all extracted data in an intermediate storage area; Clean the data in the intermediate storage area to check and correct format errors in the data, such as inconsistencies in dates or phone numbers; Then, handle missing values, which may include filling in estimated values ​​for missing data or deleting records with missing values, and then convert the cleaned data into a unified format to obtain the data to be integrated;

[0036] S2: Conduct credibility assessment on each data source to be integrated and obtain credibility scores for each data source to be integrated. This score provides a quantitative basis for the data integration system to judge the credibility of the data source and is the basis for subsequent weighted processing. For overlapping data sets from different data sources to be integrated in the data to be integrated, perform weighted processing on the overlapping data sets according to the credibility scores of each data source to be integrated to obtain weighted composite data. This step ensures that when processing overlapping data sets, information from multiple data sources can be comprehensively considered instead of simply selecting data from one data source. Use the weighted composite data to replace overlapping data sets in the data to be integrated to obtain complete data and ensure data consistency and accuracy.

[0037] S3: Define the target data model. According to the target data model, the target data model is a blueprint of the expected data structure and format after data integration; it defines the organization of the data, data types, field names and relationships, etc. The target data model is defined to ensure the consistency and comprehensibility of the data, which is convenient for subsequent data analysis and application development; standardize the complete data to eliminate the format differences between different data sources, so that the data can be uniformly processed and analyzed after integration to obtain standardized data; integrate the standardized data. The integration operation ensures the consistency and integrity of the data set, and provides a single and reliable data source for the final use of the data. A unified data set is obtained and stored in a data warehouse. A data warehouse is a system specially designed to store and manage large amounts of data. It supports efficient data query and analysis.

[0038] Specific operations for credibility assessment of each data source to be integrated:

[0039] For any data source to be integrated, sample data is extracted from the data source to represent the entire data set. The purpose of extracting sample data is to evaluate the quality of the data source without processing the entire data set. This is an effective and resource-saving method. Compare the sample data with known reliable data to calculate the error rate. Count the proportion of missing values ​​and null values ​​in the sample data and calculate the completeness rate. Data integrity is crucial for data analysis and decision-making. Missing values ​​may affect the accuracy and reliability of the analysis results. Calculate the average of the error rate and the completeness rate, and then multiply the calculated average by 100 to obtain the quality score of the data source to be integrated. Based on the update frequency of the data source to be integrated, obtain the update score of the data source to be integrated. The higher the update frequency, the higher the update score, because the update frequency directly affects the timeliness of the data, which is particularly important for business scenarios that require the latest information. Compare the historical data of the data source to be integrated with the known accurate historical data to obtain the historical quality score of the data source to be integrated. The historical quality score reflects the reliability of the data source in the past and helps to understand the overall performance and trend of the data source.

[0040] For any data source to be integrated, use the formula Calculate and obtain the credibility score R of the data source to be integrated, where A, B and c are the quality score, update score and historical quality score of the data source to be integrated respectively; , and They are the weights of quality score, update score and historical quality score respectively.

[0041] According to the comprehensive scoring results of each data source to be integrated, the specific operation of weighted processing of overlapping data is as follows: for any overlapping data set x i , i=1, 2, ..., n; where n indicates that the source of the current overlapping data set is n data sources to be integrated; using the formula Calculate and obtain the coincident data set x i The weighted sum of data X; where R i Represents the credibility score of the i-th data source to be integrated.

[0042] The specific operations for standardizing complete data include:

[0043] Determine the data type and format specification of the target field. Unified data type and format specification are the basis of data standardization and help ensure data consistency and accuracy. Create a mapping relationship document for each data source to be integrated, and record the correspondence between the fields of the data source to be integrated and the target fields. The mapping relationship document is a key reference in the data conversion process. It helps understand how data from different data sources are mapped to a unified data model. Determine the target field corresponding to each field of the data source to be integrated to ensure the accuracy of data conversion and avoid data loss or errors during the conversion process. Use the data conversion function library to apply a series of predefined data conversion functions to convert the fields in the complete data into the data types defined in the target data model according to the mapping relationship and the target data model, such as converting a string into a date format, or unifying similar data from different data sources into the same data type. According to the target data model, perform structural conversion on the complete data, including operations such as flattening nested structures and merging or splitting fields. Flattening nested structures converts nested or hierarchical data structures into a single two-dimensional table format, thereby simplifying the data structure and obtaining standardized data.

[0044] All operations on integrated data sources are stored through blockchain technology. Blockchain is a distributed ledger technology that ensures the security and immutability of data through encryption algorithms. Each block contains a series of transaction records and is connected to the previous block through a chain structure to form a growing data chain. The decentralized nature of blockchain means that there is no single control center, and data is maintained and verified by multiple nodes in the network. The introduction of smart contracts enables blockchain to automatically execute contract terms, further improving efficiency and transparency. This technology is widely used in many fields such as cryptocurrency, supply chain management, smart contracts, and identity authentication.

[0045] Specific operations for storing all operations of integrated data sources through blockchain technology:

[0046] Choose a suitable blockchain platform to store data operation logs. This platform needs to be able to meet the requirements of data security, immutability and accessibility. The data operation logs should be recorded and stored on the blockchain. The data operation logs include: creation, extraction, update and deletion operations of data in the data source to be integrated, as well as related metadata, such as the specific time of the operation, the identity of the user who performed the operation and the specific content of the data change. Recording detailed operation logs helps to track the history of data changes and increase the transparency and auditability of data operations.

[0047] A unique hash value of fixed length is generated for each data operation log using a hash algorithm. This hash value represents a summary of the log content. Any minor change to the log content will cause the hash value to change. The generated hash value is stored on the blockchain through smart contracts. Smart contracts are programs running on the blockchain that can be automatically executed when preset conditions are met. The use of smart contracts improves the automation and decentralization of the storage process, reduces human intervention, and ensures the fairness and immutability of data storage.

[0048] Set a fixed time interval. At each time interval, read the stored data operation log hash value from the blockchain and compare it with the locally generated hash value. If the two are consistent, it means that the data operation log has not been tampered with. If they are inconsistent, it means that the data operation log may have been tampered with or is incorrect, and corresponding response measures need to be executed according to the established security protocol.

[0049] Part of the data from the data source to be integrated can be stored in the distributed cache system. The specific operations are as follows:

[0050] Select a distributed cache system, which can reduce the access pressure on the main data storage and improve the data reading speed; traverse all data sources to be integrated, obtain the historical access records of the data sources to be integrated, and calculate the access frequency of the data sources to be integrated; set a high-frequency threshold, and classify the data sources to be integrated with access frequencies greater than the high-frequency threshold as high-frequency access data sources; set the data expiration time according to the update frequency of the high-frequency access data sources. The expiration time setting ensures that the data in the cache is the latest and avoids reading outdated data;

[0051] When there is a data read request, the target data is first read from the distributed cache system. This step can quickly respond to data requests. If the data is in the cache, there is no need to access the backend data warehouse, which improves the response speed. If the target data exists in the distributed cache system, the cached target data is directly returned. If the target data does not exist in the distributed cache system, the target data is read from the data warehouse and it is determined whether the source of the target data is a high-frequency access data source. If so, the read target data is stored in the distributed cache system at the same time; if not, no operation is performed.

[0052] When the cache time of cached data in the distributed cache system exceeds the data expiration time, the cached data is deleted. Regularly cleaning up expired data can free up cache space and ensure that the data stored in the cache system is active and relevant.

[0053] Embodiment 2, a multi-source heterogeneous data integration system of a data integration middleware, such as Figure 1 As shown, including:

[0054] The data extraction module is used to identify the type of data source to be integrated, and perform corresponding connection operations on each data source to be integrated according to the type of data source to be integrated; based on the data volume and update frequency of the data source to be integrated, perform full or incremental extraction operations on the data source to be integrated; store all extracted data in the intermediate storage area; clean the data in the intermediate storage area, and then convert the cleaned data into a unified format to obtain the data to be integrated;

[0055] The data processing module to be integrated is used to evaluate the credibility of each data source to be integrated, and to perform weighted processing on the overlapping data sets from different data sources to be integrated according to the credibility score of each data source to be integrated to obtain weighted combined data; and to use the weighted combined data to replace the overlapping data sets in the data to be integrated to obtain complete data;

[0056] The data storage module is used to perform standardization operations on the complete data according to the target data model to obtain standardized data; and to integrate the standardized data and store it in the data warehouse.

[0057] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention. Parts not described in detail in this specification belong to the prior art known to those skilled in the art.

Claims

1. A multi-source heterogeneous data integration method for data integration middleware, characterized in that: include: S1: Determine the data source to be integrated, identify the type of the data source to be integrated, and perform corresponding connection operations on each data source to be integrated according to the type of the data source to be integrated; Based on the data volume and update frequency of the data source to be integrated, perform full data extraction for the data source to be integrated whose data volume is less than the data volume threshold and whose update frequency is less than the frequency threshold; For the remaining data sources to be integrated, perform incremental extraction operations; store all extracted data in the intermediate storage area; clean the data in the intermediate storage area, and then convert the cleaned data into a unified format to obtain the data to be integrated; S2: Perform credibility evaluation on each data source to be integrated to obtain a credibility score for each data source to be integrated; for overlapping data sets from different data sources to be integrated in the data to be integrated, perform weighted processing on the overlapping data sets according to the credibility scores of each data source to be integrated to obtain weighted combined data; use the weighted combined data to replace the overlapping data sets in the data to be integrated to obtain complete data; S3: Define the target data model, and perform standardization operations on the complete data according to the target data model to obtain standardized data; integrate the standardized data to obtain a unified data set, and store the unified data set in the data warehouse.

2. A multi-source heterogeneous data integration method for data integration middleware according to claim 1, characterized in that: Specific operations for credibility assessment of each data source to be integrated: For any data source to be integrated, extract sample data from the data source to be integrated; compare the sample data with known reliable data to calculate the error rate; calculate the ratio of missing values ​​and null values ​​in the sample data to calculate the completeness rate; calculate the average of the error rate and the completeness rate, and then multiply the average by 100 to obtain the quality score of the data source to be integrated; Based on the update frequency of the data source to be integrated, the update score of the data source to be integrated is obtained. The higher the update frequency, the higher the update score is obtained. The historical data of the data source to be integrated is compared with the known accurate historical data to obtain the historical quality score of the data source to be integrated. For any data source to be integrated, use the formula Calculate and obtain the credibility score R of the data source to be integrated, where A, B and c are the quality score, update score and historical quality score of the data source to be integrated respectively; , and They are the weights of quality score, update score and historical quality score respectively.

3. The multi-source heterogeneous data integration method of data integration middleware according to claim 2, characterized in that: According to the comprehensive scoring results of each data source to be integrated, the specific operation of weighted processing of overlapping data is as follows: for any overlapping data set x i , i=1,2,…,n; Where n indicates that the source of the current overlapping data set is n data sources to be integrated; Using the formula Calculate and obtain the coincident data set x i The weighted sum of data X; where R i Represents the credibility score of the i-th data source to be integrated.

4. The multi-source heterogeneous data integration method of data integration middleware according to claim 3 is characterized in that: The specific operations for standardizing complete data include: determining the data type and format specification of the target field; creating a mapping relationship document for each data source to be integrated, recording the correspondence between the fields of the data source to be integrated and the target fields; determining the target field corresponding to each field of the data source to be integrated; using the data conversion function library to convert the fields in the complete data into the data types defined in the target data model according to the mapping relationship and the target data model; and performing structural conversion on the complete data according to the target data model to obtain standardized data.

5. The multi-source heterogeneous data integration method of data integration middleware according to claim 4, characterized in that: All operations on integrated data sources are stored through blockchain technology.

6. A multi-source heterogeneous data integration method for data integration middleware according to claim 5, characterized in that: The specific operations of storing all operations on the integrated data source through blockchain technology are as follows: select a blockchain platform to record and store data operation logs on the blockchain. The data operation logs include: creation, extraction, update and deletion operations of data in the data source to be integrated, as well as the specific time of the operation, the identity of the user who performed the operation and the specific changes in the data; generate a hash value for each data operation log, and store the generated hash value on the blockchain through a smart contract; set a fixed time interval, and read the stored data operation log hash value from the blockchain at each time interval, and compare it with the locally generated hash value; if the two are consistent, it means that the data operation log has not been tampered with; if they are inconsistent, it means that the data operation log may be tampered with or is incorrect, and response measures need to be implemented.

7. A multi-source heterogeneous data integration method for data integration middleware according to claim 6, characterized in that: Part of the data of the data source to be integrated can be stored through the distributed cache system. The specific operations are as follows: select the distributed cache system; traverse all the data sources to be integrated, obtain the historical access records of the data sources to be integrated, and calculate the access frequency of the data sources to be integrated; set a high-frequency threshold, and classify the data sources to be integrated whose access frequency is greater than the high-frequency threshold as high-frequency access data sources; set the data expiration time according to the update frequency of the high-frequency access data source; when there is a data read request, first read the target data from the distributed cache system; if the target data exists in the distributed cache system, directly return the cached target data; if the target data does not exist in the distributed cache system, read the target data from the data warehouse, and determine whether the source of the target data is a high-frequency access data source. If so, store the read target data in the distributed cache system at the same time; If not, no operation is performed; when the cache time of the cached data in the distributed cache system exceeds the data expiration time, the cached data is deleted.

8. A multi-source heterogeneous data integration system of data integration middleware, characterized in that: The system is applied to a multi-source heterogeneous data integration method of a data integration middleware as described in any one of claims 1 to 7 above, comprising: a data extraction module, used to identify the type of data source to be integrated, and perform corresponding connection operations on each data source to be integrated according to the type of data source to be integrated; based on the data volume and update frequency of the data source to be integrated, perform full or incremental extraction operations on the data source to be integrated; store all extracted data in an intermediate storage area; clean the data in the intermediate storage area, and then convert the cleaned data into a unified format to obtain the data to be integrated; a data processing module to be integrated, used to perform credibility assessment on each data source to be integrated, and perform weighted processing on overlapping data sets from different data sources to be integrated according to the credibility score of each data source to be integrated to obtain weighted composite data; use the weighted composite data to replace the overlapping data sets in the data to be integrated to obtain complete data; a data storage module, used to perform standardization operations on the complete data according to the target data model to obtain standardized data; and store the standardized data in a data warehouse after integration.

Images