Data processing method and system based on multi-source heterogeneous data
By generating unified intermediate representation data blocks and using intelligent pattern mapping technology, the problems of poor correlation and real-time performance of multi-source heterogeneous data are solved, enabling efficient and flexible data fusion and processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN HUASI ZHIYUAN TECH CO LTD
- Filing Date
- 2026-02-10
- Publication Date
- 2026-05-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing technologies, when processing multi-source heterogeneous data, suffer from poor correlation and difficulty in effective integration due to independent data storage and a lack of a unified semantic model. Furthermore, traditional ETL processes cannot adapt to rapid changes in data sources and real-time requirements.
A unified intermediate representation data block is generated through a configurable adapter, combined with a machine learning model for data quality exploration and repair, semantic alignment and entity association are performed using a knowledge graph, and computing resources are dynamically scheduled to achieve data fusion and output.
It significantly reduces the complexity and manpower costs of data access and integration, improves the flexibility and maintainability of the system, ensures data quality and consistency, and supports rapid response to changes in data sources.
Smart Images
Figure CN122086994A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically, to a data processing method and system based on multi-source heterogeneous data. Background Technology
[0002] In today's era of big data and artificial intelligence, data has become a core production factor. The data environment of enterprises and institutions is becoming increasingly complex, and data sources exhibit typical characteristics of "multi-source heterogeneity." "Multi-source" is reflected in the fact that data comes from various sources such as internal business systems (e.g., CRM, ERP), IoT sensors, social media, public networks, and third-party data service providers. "Heterogeneity" is manifested in the huge differences in data structure, format, protocol, and quality, including structured data (e.g., database tables), semi-structured data (e.g., JSON, XML logs), and unstructured data (e.g., text, images, videos). Existing technologies typically face the following challenges when processing multi-source heterogeneous data: data from different sources are stored independently, employing different data models and access interfaces, lacking a unified semantic model, resulting in poor correlation between data and difficulty in effective correlation analysis and fusion; traditional ETL (Extract, Transform, Load) processes are usually hard-coded and batch-processed, unable to adapt to rapid changes in data sources and real-time requirements. Therefore, we propose an improvement, namely a data processing method and system based on multi-source heterogeneous data. Summary of the Invention
[0003] This invention provides a data processing method based on multi-source heterogeneous data, including: S1. Extract data from multiple heterogeneous data sources through a configurable adapter and generate a unified intermediate representation data block; S2. Based on a predefined rule base and machine learning model, perform data quality exploration, verification and repair on the intermediate representation data block, and generate a quality assessment report; S3. Based on the target data model, perform pattern mapping, semantic alignment, and entity association on the quality-processed data to complete the fusion of multi-source data; S4. Based on the pre-declared or user-defined data processing task description, perform the transformation, calculation and aggregation operations of the merged data under the unified resource scheduling framework; S5. Output the processing results to the specified target data storage or application service in a predetermined format.
[0004] As a preferred technical solution of this application, in S1, the configurable adapter supports at least one of structured query interface, file transfer protocol, message queue and streaming data interface; the unified intermediate representation data block is encapsulated using a self-describing unified serialization format.
[0005] As a preferred technical solution of this application, in S2, the data quality exploration includes the measurement of data integrity, consistency, accuracy and timeliness; the machine learning model is used to identify abnormal patterns and predict and fill missing values in the data.
[0006] As a preferred technical solution of this application, the comprehensive quality score Q of data quality exploration is calculated by the following formula to drive differentiated repair strategies; ; in, For the quantity of quality dimensions, This represents the individual score for the i-th quality dimension, with a value within the range [0,1]. Let i be the dynamic weight of the i-th quality dimension, satisfying Its value is dynamically adjusted based on the downstream application type or user configuration; P is the penalty coefficient for data anomalies; P=λ N, where N is the number of key anomalies identified by the machine learning model, and λ is the preset unit penalty value.
[0007] As a preferred technical solution of this application, in S3, semantic alignment utilizes knowledge graphs or domain ontology libraries to identify and associate fields representing the same real-world entity in different data sources.
[0008] As a preferred technical solution of this application, in S4, the unified resource scheduling framework dynamically allocates tasks to batch processing engines, stream processing engines or machine learning frameworks for execution based on the computational characteristics of the tasks, and monitors the task execution status and resource consumption.
[0009] A data processing system based on multi-source heterogeneous data, used to implement a data processing method based on multi-source heterogeneous data, includes: The data access and buffering module is used to connect to multiple heterogeneous data sources and generate a unified intermediate representation data block. The data quality governance module is used for quality assessment and repair of data blocks; The data fusion and modeling module is used to achieve semantic alignment, model mapping, and entity association of data; The task scheduling and computing engine module is used to parse and process tasks, schedule computing resources, and execute computing logic. The data service and output module is used to manage the storage and access interfaces of the processing results.
[0010] As a preferred technical solution of this application, it also includes: The metadata management module is used to centrally manage metadata across the entire chain, from data sources and processing to output results, and to provide support for data lineage tracing, impact analysis, and compliance auditing.
[0011] As a preferred technical solution of this application, it also includes a visual configuration management interface for configuring data source connections, defining data processing pipelines, setting quality rules, and monitoring system operation status.
[0012] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a data processing method based on multi-source heterogeneous data.
[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: In the scheme of this application: 1. This application, through unified data model abstraction and intelligent pattern mapping technology, can automatically discover, align and integrate data from different sources and with different structures, significantly reducing the complexity and human cost of data access and fusion, and breaking down data silos; 2. This application adopts a declarative data processing pipeline definition and a metadata-based driven execution mechanism, which makes the data processing process configurable and reusable. When the data source or business logic changes, there is no need to rewrite a lot of code. Only the configuration needs to be adjusted to respond quickly, which greatly improves the flexibility and maintainability of the system. 3. This application incorporates a pluggable and configurable data quality rule engine and introduces a dynamic quality assessment model based on statistical learning and anomaly detection. The system can automatically identify common and specific quality problems in the data, providing quality reports and remediation suggestions, thereby ensuring the consistency and reliability of data input to downstream applications. Attached Figure Description
[0014] Figure 1 A flowchart of the data processing method based on multi-source heterogeneous data provided in this application; Figure 2 A schematic diagram of the data processing system based on multi-source heterogeneous data provided in this application. Detailed Implementation
[0015] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0016] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the embodiments of the present invention can be combined with each other.
[0017] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.
[0018] Example 1, please refer to Figure 1 A data processing method based on multi-source heterogeneous data, comprising: S1. Through a configurable adapter, data is extracted from multiple heterogeneous data sources and a unified intermediate representation data block is generated. The interface protocols and data formats of heterogeneous data sources (such as databases, files, message queues, etc.) vary greatly. Traditional fixed adapters need to be developed separately for each data source. The configurable adapter supports flexible configuration of access parameters, eliminating the need for repeated development. At the same time, the unified intermediate representation data block eliminates the dependence of subsequent processing steps on the original data format, making the data processing flow more universal. S2. Based on a predefined rule base and machine learning model, the intermediate representation data block is subjected to data quality exploration, verification, and repair, and a quality assessment report is generated. The predefined rule base can cover known data quality problems (such as format errors and logical conflicts) to achieve rapid verification. The machine learning model can identify unknown abnormal patterns and predict and fill missing values, making up for the limitations of the rule base. The quality assessment report makes the data quality status transparent, which is convenient for users to handle in a targeted manner and avoid decision-making errors or system failures caused by low-quality data. S3. Based on the target data model, perform schema mapping, semantic alignment, and entity association on the quality-processed data to complete the fusion of multi-source data. Multi-source data often have problems such as schema heterogeneity (e.g., different field names) and semantic ambiguity (e.g., different expressions of the same concept). Schema mapping solves structural differences, and semantic alignment and entity association eliminate semantic ambiguity, allowing related data scattered in different data sources to form a complete information set, supporting more comprehensive data analysis and application. S4. Based on the pre-declared or user-defined data processing task description, and within the unified resource scheduling framework, perform transformation, computation, and aggregation operations on the fused data. The unified resource scheduling framework can dynamically allocate suitable computing engines based on task computation characteristics (such as batch processing, stream processing, and machine learning computation), avoiding resource waste. It also supports user-defined task descriptions, allowing adaptation to different scenarios (such as report statistics, real-time analysis, and model training) without modifying the underlying code, thus improving the flexibility and scalability of the solution. S5. Output the processing results to the specified target data storage or application service in a predetermined format. The predetermined format (such as JSON, Parquet, database table structure) conforms to the access standards of mainstream storage and applications. The specified output destination can be directly connected to the downstream system without additional data format conversion or transmission adaptation work, thereby improving the efficiency of the entire data processing chain.
[0019] Furthermore, in S1, the configurable adapter supports at least one of the following: structured query interface, file transfer protocol, message queue, and streaming data interface; the unified intermediate representation data block is encapsulated using a self-describing unified serialization format; support for multiple interface protocols means that it can cover most mainstream data sources (such as SQL interface for databases, FTP / SFTP protocol for files, and Kafka message queue for real-time data), meeting the needs of multi-source data access in complex scenarios; the self-describing serialization format (such as Protobuf, JSON Schema) contains data structure description information, which can be parsed without additional documentation, while being compatible with different programming languages and processing frameworks, improving the universality of data transmission and processing.
[0020] Furthermore, in S2, the data quality inspection includes measurements of data integrity, consistency, accuracy, and timeliness; the machine learning model is used to identify abnormal patterns and predict missing values in the data; integrity, consistency, accuracy, and timeliness are the core dimensions of data quality, and comprehensive measurement can ensure that the data meets the standards in each key indicator; the machine learning model identifies anomalies and predicts missing values based on data distribution patterns and historical characteristics, which is more efficient than manual completion, and the completion results are more in line with the actual data patterns, avoiding bias caused by subjective judgment.
[0021] Furthermore, the overall quality score Q of the data quality exploration is calculated using the following formula to drive differentiated remediation strategies; ; in, For the quantity of quality dimensions, This represents the individual score for the i-th quality dimension, with a value within the range [0,1]. Let i be the dynamic weight of the i-th quality dimension, satisfying Its value is dynamically adjusted based on the downstream application type or user configuration; P is the penalty coefficient for data anomalies; P=λ N, where N is the number of key anomalies identified by the machine learning model, and λ is a preset unit penalty value. This enables refined and personalized data repair, improving repair efficiency and reducing unnecessary repair costs.
[0022] Furthermore, in S3, semantic alignment utilizes knowledge graphs or domain ontology libraries to identify and associate fields representing the same real-world entity in different data sources. Knowledge graphs / domain ontology libraries contain structured knowledge such as concept definitions and entity relationships within the domain, providing a unified semantic standard to quickly identify fields with "synonymous but different names" in different data sources (such as "customer ID" and "user number"). Compared to traditional alignment methods based on string matching, it has higher accuracy and can handle complex semantic association scenarios (such as indirectly associated entity fields).
[0023] Furthermore, in S4, the unified resource scheduling framework dynamically allocates tasks to batch processing engines, stream processing engines, or machine learning frameworks based on their computational characteristics, and monitors task execution status and resource consumption. Different computing engines are adapted to different task types (batch processing engines are suitable for massive offline computing, stream processing engines are suitable for real-time data processing, and machine learning frameworks are suitable for model training). Dynamic allocation allows tasks to be executed on the optimal engine, improving processing efficiency. Real-time monitoring of task status and resource consumption can promptly detect issues such as task failures and resource bottlenecks, facilitating rapid troubleshooting and preventing system congestion caused by excessive resource consumption.
[0024] Example 2, as Figure 2 As shown, a data processing system based on multi-source heterogeneous data is used to implement a data processing method based on multi-source heterogeneous data, including: The data access and buffering module is used to connect to multiple heterogeneous data sources and generate a unified intermediate representation data block. The data access function is separated into an independent module, which facilitates the separate upgrade and maintenance of the data source adaptation logic in the future. When adding a new data source, only the module configuration needs to be extended, without modifying other system components. The buffering mechanism can cope with sudden high-concurrency data transmission from the data source, avoid data loss or system overload, and ensure the continuity of data access. The data quality governance module is used to assess and repair the quality of data blocks. The independent quality governance module integrates the logic of quality detection, verification and repair, avoiding the management chaos caused by the quality control logic being scattered in various processing links. The data fusion and modeling module is used to achieve semantic alignment, model mapping, and entity association of data; it makes the data fusion logic clearer, easier to maintain and optimize, and improves the accuracy and standardization of data fusion. The task scheduling and computing engine module is used to parse and process tasks, schedule computing resources and execute computing logic; improve the orderliness and efficiency of task execution, realize on-demand allocation of computing resources, and reduce system operating costs; The data service and output module manages the storage and access interfaces for processing results; it simplifies the management and access process of processing results and improves the availability and security of data services.
[0025] Furthermore, it also includes: The metadata management module is used to centrally manage end-to-end metadata from data sources and processing processes to output results, and provides support for data lineage tracing, impact analysis, and compliance auditing. End-to-end metadata records information such as data source, processing flow, format conversion, and access records. Data lineage tracing can quickly locate the root cause of data problems (such as tracing back to the original data source or processing step when an output result is incorrect). Impact analysis can predict the impact of changes in data sources or processing rules on downstream processes, facilitating operational decisions. Compliance auditing meets data security regulatory requirements and avoids compliance risks caused by opaque data flow.
[0026] Furthermore, it also includes a visual configuration management interface for configuring data source connections, defining data processing pipelines, setting quality rules, and monitoring system operation status. The visual interface allows users to complete core configurations (such as setting data source connection parameters and dragging and dropping processing steps) without writing code, and can be operated by non-technical personnel. Real-time monitoring of system operation status allows users to intuitively grasp task progress, resource usage, data quality, etc., and quickly discover and respond to anomalies (such as task failure or data quality failure).
[0027] Example 3: A computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a data processing method based on multi-source heterogeneous data.
[0028] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, an electrical connection, or a connection that allows communication between them; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0029] Obviously, the embodiments described above are merely some embodiments of the present invention, not all embodiments. The accompanying drawings show preferred embodiments of the present invention, but do not limit the patent scope of the present invention. The present invention can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the patent protection scope of this invention.
Claims
1. A data processing method based on multi-source heterogeneous data, characterized in that, include: S1. Extract data from multiple heterogeneous data sources through a configurable adapter and generate a unified intermediate representation data block; S2. Based on a predefined rule base and machine learning model, perform data quality exploration, verification and repair on the intermediate representation data block, and generate a quality assessment report; S3. Based on the target data model, perform pattern mapping, semantic alignment, and entity association on the quality-processed data to complete the fusion of multi-source data; S4. Based on the pre-declared or user-defined data processing task description, perform the transformation, calculation and aggregation operations of the merged data under the unified resource scheduling framework; S5. Output the processing results to the specified target data storage or application service in a predetermined format.
2. The data processing method based on multi-source heterogeneous data according to claim 1, characterized in that, In S1, the configurable adapter supports at least one of the structured query interface, file transfer protocol, message queue and streaming data interface; the unified intermediate representation data block is encapsulated in a self-describing unified serialization format.
3. The data processing method based on multi-source heterogeneous data according to claim 1, characterized in that, In S2, the data quality exploration includes measurements of data integrity, consistency, accuracy, and timeliness; the machine learning model is used to identify abnormal patterns in the data and predict and impute missing values.
4. The data processing method based on multi-source heterogeneous data according to claim 3, characterized in that, The overall quality score Q of data quality exploration is calculated using the following formula to drive differentiated remediation strategies; ； in, For the quantity of quality dimensions, This represents the individual score for the i-th quality dimension, with a value within the range [0,1]. Let i be the dynamic weight of the i-th quality dimension, satisfying Its value is dynamically adjusted based on the downstream application type or user configuration; P is the penalty coefficient for data anomalies; P=λ N, where N is the number of key anomalies identified by the machine learning model, and λ is the preset unit penalty value.
5. The data processing method based on multi-source heterogeneous data according to claim 1, characterized in that, In S3, semantic alignment utilizes knowledge graphs or domain ontology libraries to identify and associate fields representing the same real-world entity from different data sources.
6. The data processing method based on multi-source heterogeneous data according to claim 1, characterized in that, In S4, the unified resource scheduling framework dynamically allocates tasks to batch processing engines, stream processing engines, or machine learning frameworks based on their computational characteristics, and monitors task execution status and resource consumption.
7. A data processing system based on multi-source heterogeneous data, used to implement the data processing method based on multi-source heterogeneous data as described in any one of claims 1-6, characterized in that, include: The data access and buffering module is used to connect to multiple heterogeneous data sources and generate a unified intermediate representation data block. The data quality governance module is used for quality assessment and repair of data blocks; The data fusion and modeling module is used to achieve semantic alignment, model mapping, and entity association of data; The task scheduling and computing engine module is used to parse and process tasks, schedule computing resources, and execute computing logic. The data service and output module is used to manage the storage and access interfaces of the processing results.
8. The data processing system based on multi-source heterogeneous data according to claim 7, characterized in that, Also includes: The metadata management module is used to centrally manage metadata across the entire chain, from data sources and processing to output results, and to provide support for data lineage tracing, impact analysis, and compliance auditing.
9. The data processing system based on multi-source heterogeneous data according to claim 7, characterized in that, It also includes a visual configuration management interface for configuring data source connections, defining data processing pipelines, setting quality rules, and monitoring system operation status.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the data processing method based on multi-source heterogeneous data as described in any one of claims 1-6.