Data island resolution and integration system in data-in-station network

Abstract

The invention provides a data island resolution and integration system in a station network in data, which relates to the field of electric digital data processing and comprises a data access acquisition module, a data standardization management module, a data fusion association module and a data security management and control module. The data access acquisition module gets through data input channels of different sources and realizes efficient convergence of diversified data, and the data standardization management module is used for cleaning, standardizing and quality control of access data and eliminating semantic and structural differences. The data fusion association module realizes deep integration of cross-source data through identification, association and fusion technologies, and the data security management and control module is used for performing secure and controllable open service on the integrated data. According to the system, the data island barrier is effectively broken, safe and controllable integration and service of cross-source data are realized, and a complete technical solution is provided for construction of a data platform.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of digital data processing, in particular to a data island elimination and integration system in a data middle platform. BACKGROUND

[0002] With the deepening of information construction, enterprises and organizations have accumulated a large amount of business data. These data are scattered in different business systems, databases and application platforms, forming a large number of isolated data islands. The problem of data islands seriously hinders the circulation and value mining of data, resulting in that data resources cannot be uniformly managed and comprehensively utilized. As a new type of data architecture, the data middle platform emerges as the times require, and its core goal is to break down the barriers of data islands and realize cross-system and cross-platform data integration and service. However, due to the significant differences in data format, transmission protocol, coding standard, semantic definition and other aspects of different data sources, and the uneven data quality, how to efficiently eliminate data islands and realize deep integration has become a key technical problem faced by the construction of data middle platform.

[0003] In the field of data integration, the existing technology mainly adopts three kinds of algorithms to solve the matching and fusion problem of cross-source data. The first is a rule-based matching algorithm. This algorithm compares and associates data through a predefined rule set, such as judging whether the records in different data sources represent the same entity through field name, data type, business logic and other rules. The rule-based method is simple to implement and has high execution efficiency, and can quickly complete the matching task in the scene where the data mode is relatively stable. However, the main disadvantage of this method is that the formulation of rules requires a lot of manual participation and is highly dependent on domain expert knowledge. When facing heterogeneous data sources and complex business scenarios, the rule maintenance cost is extremely high, and it is difficult to adapt to the dynamic changes of data mode, lacking self-adaptive ability.

[0004] The second is a similarity calculation algorithm based on machine learning. This kind of algorithm extracts data features and trains a matching model using supervised learning or unsupervised learning methods. Common techniques include classifiers based on support vector machines, random forests, neural networks, etc. to determine whether a record pair matches. The machine learning method can automatically learn the matching pattern from the data, and has stronger generalization ability and adaptability than the rule method, and can handle more complex data relationships. However, the disadvantage of this method is that a large number of high-quality labeled samples are needed for model training, the labeling cost is high, and the model has poor interpretability, making it difficult to trace the cause when the matching result is wrong. In addition, the features of a single data source are often insufficient to fully describe the entity features, resulting in limited matching accuracy.

[0005] The third type is a graph-based propagation matching algorithm. This algorithm constructs data records and their relationships into a graph structure, and uses a propagation mechanism on the graph to utilize neighborhood information to assist in entity matching. The propagation algorithm can perform global optimization using the relationships between data, significantly improving matching results in scenarios with dense data relationship networks. However, traditional graph propagation algorithms suffer from high computational complexity; when the data scale reaches millions or more, the computational efficiency drops sharply, and the propagation process is easily affected by noisy data, leading to incorrect propagation and impacting the overall matching quality. Furthermore, this method requires a high-quality graph structure; the accuracy of the initial relationship graph directly affects the subsequent propagation results.

[0006] Regarding patented technologies for data integration systems, US Patent US08255516B1 discloses a server integration method. This patent provides a technical solution for optimizing server resource configuration by accessing performance data from multiple source servers. The method includes receiving multiple merging parameters for the desired target server, receiving the selected configuration of the new target server, calculating the minimum number of target servers required to merge multiple source servers based on performance data, selecting configurations and multiple merging parameters for the desired target server, and deriving a binning optimization solution for server merging based on performance data, the selected configuration of the new target server, and the minimum number of one or more performance indicators. However, this patent has the following shortcomings: First, it focuses on the integration of server hardware resources rather than deep data-level integration, lacking semantic understanding and content integration capabilities for heterogeneous data sources, and failing to solve the logical correlation problem of data silos. Second, the integration strategy of this patent is mainly based on performance indicators and resource utilization, without considering key data-level factors such as data quality characteristics, timeliness, and reliability, resulting in an integration solution that cannot meet the data quality and consistency requirements of a data platform.

[0007] Chinese patent CN117370325B discloses a data platform system based on big data acquisition and analysis. This patent relates to the field of data communication and processing. Its technical solution includes a data acquisition module for connecting multiple isolated databases and collecting data, a data cleaning module for cleaning the acquired data, a data integration module for integrating the cleaned multi-source data, and a server for storing the integrated data. This patent implements the basic process from data acquisition to storage through a modular architecture design, providing a solution for initially breaking down data silos. However, this patent has significant shortcomings. First, its data acquisition module lacks protocol adaptation capabilities for heterogeneous data sources and a real-time streaming acquisition mechanism, making it unable to effectively handle different transmission protocols and high-concurrency acquisition scenarios. Second, the data cleaning module's function is relatively simple, lacking in-depth governance aspects such as data standardization and quality assessment, and cannot systematically eliminate semantic and structural differences between multi-source data. Third, the data integration module lacks cross-source entity recognition and semantic association technologies, only achieving physical-level data integration without establishing logical relationships between data, resulting in insufficient integration depth. Summary of the Invention

[0008] The purpose of this invention is to address the existing shortcomings by proposing a system for resolving and integrating data silos in a data middleware network.

[0009] The present invention adopts the following technical solution:

[0010] A system for eliminating and integrating data silos in a data middleware network includes a data access and acquisition module, a data standardization and governance module, a data fusion and association module, and a data security management and control module.

[0011] The data access and acquisition module opens up data input channels from different sources and achieves efficient aggregation of diverse data. The data standardization and governance module is used to clean, standardize, and control the quality of the accessed data and eliminate semantic and structural differences. The data fusion and association module achieves deep integration of cross-source data through identification, association, and fusion technologies. The data security management and control module is used to provide secure and controllable open services for the integrated data.

[0012] The data access and acquisition module includes a data source access unit, a protocol parsing unit, and a real-time acquisition unit. The data source access unit is responsible for establishing connections with business systems, third-party platforms, sensor devices, etc., to ensure that various data sources can be smoothly accessed by the system. The protocol parsing unit performs unified parsing of different transmission protocols used by heterogeneous data sources to eliminate differences at the protocol level. The real-time acquisition unit performs real-time streaming acquisition of business data that requires rapid response, such as transaction data and log data.

[0013] The data standardization and governance module includes a data cleaning unit, a data standardization unit, and a data quality assessment unit. The data cleaning unit detects and processes data quality issues, the data standardization unit maps data content from different systems to a unified data standard model, and the data quality assessment unit establishes a quality indicator system and automatically assesses and labels the governed data.

[0014] The data fusion and association module includes a cross-source entity recognition unit, a semantic association mapping unit, and a data fusion rule engine unit. The cross-source entity recognition unit identifies records representing the same real entity scattered across different data sources and provides a basis for data association. The semantic association mapping unit establishes a semantic relationship graph of cross-source data and constructs a global data association network. The data fusion rule engine unit automatically performs data merging and conflict resolution according to preset business rules and generates a unified data view.

[0015] The data service and security management module includes a unified data service interface unit, an access control unit, and a data lineage tracing unit. The unified data service interface unit provides a standardized API gateway and data service catalog and encapsulates the underlying complex data acquisition logic. The access control unit implements fine-grained data access permission management and dynamically controls the data access scope. The data lineage tracing unit records the entire lifecycle of data from collection, governance, integration to use, tracking the source, processing, and usage destination of each piece of data.

[0016] Furthermore, the cross-source entity recognition unit includes a feature extraction processor, a similarity calculation processor, and an entity matching processor. The feature extraction processor extracts key features for entity recognition from records from different data sources and constructs entity feature vectors. The similarity calculation processor calculates the similarity between records from different data sources and generates a similarity matrix. The entity matching processor determines which records represent the same real entity based on the similarity calculation results and performs entity merging, assigning a unique global identifier to each real entity.

[0017] Furthermore, the similarity calculation processor calculates the enhanced similarity between the two data points according to the following formula:

[0018] ;

[0019] Among them, S enh (i, j) represents the enhanced similarity between record i from data source A and record j from data source B, where n is the total number of feature dimensions, and w k f represents the weight coefficient of the k-th dimension feature. ik and f jkLet sim represent the feature values ​​of record i and record j in the k-th feature dimension, respectively. k Let C be the basic similarity calculation function for the k-th dimension feature. k Cross-source complementarity of the k-th dimension feature, I is the complementarity gain coefficient. i and I j These are the sets of non-empty feature dimension indices for records i and j, respectively. This is the feature coverage consistency gain coefficient.

[0020] Furthermore, the entity matching processor calculates the propagation enhancement matching score M for record i and record j according to the following formula. prop (i, j):

[0021] ;

[0022] Where H is the maximum propagation hop count, N is the propagation attenuation factor. h (i) and N h (j) represent the h-hop neighbor set of record i and record j, respectively, w ip and w jq Record the strength weights of the relationships between i and its neighboring node p, and between j and its neighboring node q, respectively.

[0023] When the propagation enhancement matching score exceeds the matching threshold, the entity matching processor will merge record i and record j into the same entity.

[0024] Furthermore, the data fusion rule engine unit includes a rule configuration manager, a conflict detection processor, and a fusion decision processor. The rule configuration manager provides functions for defining, storing, and managing fusion rules. The conflict detection processor identifies data conflicts of the same entity in different data sources during the data fusion process and marks the conflict type and conflict field. The fusion decision processor automatically adjudicates the detected conflicts according to preset fusion rules and forms a unified data view.

[0025] The fusion decision processor calculates the fusion value V of the m records merged into the same entity on the numerical attribute according to the following formula. fus Perform the calculation:

[0026] ;

[0027] Among them, V t T represents the value of the t-th record on this attribute. t Let t be the credibility weight of the data source for the t-th record. Let be the time difference of the t-th record. This is the time-related decay coefficient. The entropy sensitivity coefficient, Here, Hnorm is the median weighting coefficient, Hnorm is the normalized information entropy, and median{} is the median function.

[0028] The beneficial effects achieved by this invention are:

[0029] This system proposes an enhanced similarity calculation model. By introducing cross-source complementarity and feature coverage consistency gain coefficients, this model fully utilizes the complementary information of multi-source data. Compared with traditional similarity calculation methods, it can more accurately identify cross-source entities, significantly improving the accuracy and recall of entity matching. The propagation-enhanced matching algorithm uses a multi-hop neighborhood propagation mechanism to perform global optimization based on the correlation between data, effectively solving the problem of inaccurate matching by simply relying on recording its own features. The improvement in matching performance is particularly significant in scenarios with dense data relationship networks. The multi-factor fusion decision model comprehensively considers multiple dimensions such as data source credibility, data timeliness, and data consistency. Through mathematical modeling, it achieves intelligent resolution of data conflicts. Compared with simple timestamp strategies or weighted averaging methods, the fusion results are more accurate and reliable.

[0030] To further understand the features and technical content of the present invention, please refer to the following detailed description and accompanying drawings. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall structural framework of the present invention;

[0032] Figure 2 This is a schematic diagram of the data access and acquisition module of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating the structure of the data standardization and governance module of this invention;

[0034] Figure 4 This is a schematic diagram illustrating the structure of the data fusion and association module of the present invention;

[0035] Figure 5 This is a schematic diagram illustrating the structure of the data service and security management module of the present invention;

[0036] Figure 6 This is a schematic diagram comparing the entity recognition performance of the present invention and traditional methods;

[0037] Figure 7 This diagram illustrates the comparison of data processing efficiency between the present invention and traditional solutions. Detailed Implementation

[0038] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the spirit of the present invention. Furthermore, the accompanying drawings of the present invention are for simple illustrative purposes only and are not depictions of actual dimensions; this is stated beforehand. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of protection of the present invention.

[0039] Example 1.

[0040] This embodiment provides a system for resolving and integrating data silos in a data middleware network, combined with... Figure 1 It includes a data access and acquisition module, a data standardization and governance module, a data fusion and association module, and a data security management and control module.

[0041] The data access and acquisition module opens up data input channels from different sources and achieves efficient aggregation of diverse data. The data standardization and governance module is used to clean, standardize, and control the quality of the accessed data and eliminate semantic and structural differences. The data fusion and association module achieves deep integration of cross-source data through identification, association, and fusion technologies. The data security management and control module is used to provide secure and controllable open services for the integrated data.

[0042] The data access and acquisition module includes a data source access unit, a protocol parsing unit, and a real-time acquisition unit. The data source access unit is responsible for establishing connections with business systems, third-party platforms, sensor devices, etc., to ensure that various data sources can be smoothly accessed by the system. The protocol parsing unit performs unified parsing of different transmission protocols used by heterogeneous data sources to eliminate differences at the protocol level. The real-time acquisition unit performs real-time streaming acquisition of business data that requires rapid response, such as transaction data and log data.

[0043] The data standardization and governance module includes a data cleaning unit, a data standardization unit, and a data quality assessment unit. The data cleaning unit detects and processes data quality issues, the data standardization unit maps data content from different systems to a unified data standard model, and the data quality assessment unit establishes a quality indicator system and automatically assesses and labels the governed data.

[0044] The data fusion and association module includes a cross-source entity recognition unit, a semantic association mapping unit, and a data fusion rule engine unit. The cross-source entity recognition unit identifies records representing the same real entity scattered across different data sources and provides a basis for data association. The semantic association mapping unit establishes a semantic relationship graph of cross-source data and constructs a global data association network. The data fusion rule engine unit automatically performs data merging and conflict resolution according to preset business rules and generates a unified data view.

[0045] The data service and security management module includes a unified data service interface unit, an access control unit, and a data lineage tracing unit. The unified data service interface unit provides a standardized API gateway and data service catalog and encapsulates the underlying complex data acquisition logic. The access control unit implements fine-grained data access permission management and dynamically controls the data access scope. The data lineage tracing unit records the entire lifecycle of data from collection, governance, integration to use, tracking the source, processing, and usage destination of each piece of data.

[0046] The cross-source entity recognition unit includes a feature extraction processor, a similarity calculation processor, and an entity matching processor. The feature extraction processor extracts key features for entity recognition from records from different data sources and constructs entity feature vectors. The similarity calculation processor calculates the similarity between records from different data sources and generates a similarity matrix. The entity matching processor determines which records represent the same real entity based on the similarity calculation results and performs entity merging, assigning a unique global identifier to each real entity.

[0047] The similarity calculation processor calculates the enhanced similarity between the two data points according to the following formula:

[0048] ;

[0049] Among them, S enh (i, j) represents the enhanced similarity between record i from data source A and record j from data source B, where n is the total number of feature dimensions, and w k f represents the weight coefficient of the k-th dimension feature. ik and f jk Let sim represent the feature values ​​of record i and record j in the k-th feature dimension, respectively. k Let C be the basic similarity calculation function for the k-th dimension feature. k Cross-source complementarity of the k-th dimension feature, I is the complementarity gain coefficient. i and I j These are the sets of non-empty feature dimension indices for records i and j, respectively. This is the feature coverage consistency gain coefficient.

[0050] The entity matching processor calculates the propagation-enhanced matching score M for record i and record j according to the following formula. prop (i, j):

[0051] ;

[0052] Where H is the maximum propagation hop count, N is the propagation attenuation factor. h (i) and N h (j) represent the h-hop neighbor set of record i and record j, respectively, w ip and w jq Record the strength weights of the relationships between i and its neighboring node p, and between j and its neighboring node q, respectively.

[0053] When the propagation enhancement matching score exceeds the matching threshold, the entity matching processor will merge record i and record j into the same entity.

[0054] The data fusion rule engine unit includes a rule configuration manager, a conflict detection processor, and a fusion decision processor. The rule configuration manager provides functions for defining, storing, and managing fusion rules. The conflict detection processor identifies data conflicts of the same entity in different data sources during the data fusion process and marks the conflict type and conflict field. The fusion decision processor automatically adjudicates the detected conflicts according to preset fusion rules and forms a unified data view.

[0055] The fusion decision processor calculates the fusion value V of the m records merged into the same entity on the numerical attribute according to the following formula. fus Perform the calculation:

[0056] ;

[0057] Among them, V t T represents the value of the t-th record on this attribute. t Let t be the credibility weight of the data source for the t-th record. Let be the time difference of the t-th record. This is the time-related decay coefficient. The entropy sensitivity coefficient, Here, Hnorm is the median weighting coefficient, Hnorm is the normalized information entropy, and median{} is the median function.

[0058] Example 2.

[0059] This embodiment includes all the contents of Embodiment 1, and provides a system for eliminating and integrating data silos in a data middleware network, including a data access and acquisition module, a data standardization and governance module, a data fusion and association module, and a data security management and control module.

[0060] The data access and acquisition module opens up data input channels from different sources and achieves efficient aggregation of diverse data. The data standardization and governance module is used to clean, standardize, and control the quality of the accessed data and eliminate semantic and structural differences. The data fusion and association module achieves deep integration of cross-source data through identification, association, and fusion technologies. The data security management and control module is used to provide secure and controllable open services for the integrated data.

[0061] Combination Figure 2 The data access and acquisition module includes a data source access unit, a protocol parsing unit, and a real-time acquisition unit. The data source access unit is responsible for establishing connections with business systems, third-party platforms, sensor devices, etc., to ensure that various data sources can be smoothly accessed by the system. The protocol parsing unit performs unified parsing of different transmission protocols used by heterogeneous data sources to eliminate differences at the protocol level. The real-time acquisition unit performs real-time streaming acquisition of business data that requires rapid response, such as transaction data and log data.

[0062] Combination Figure 3 The data standardization and governance module includes a data cleaning unit, a data standardization unit, and a data quality assessment unit. The data cleaning unit detects and processes data quality issues. The data standardization unit maps data content from different systems to a unified data standard model. The data quality assessment unit establishes a quality indicator system and automatically assesses and labels the governed data.

[0063] Combination Figure 4 The data fusion and association module includes a cross-source entity recognition unit, a semantic association mapping unit, and a data fusion rule engine unit. The cross-source entity recognition unit identifies records representing the same real entity scattered across different data sources and provides a basis for data association. The semantic association mapping unit establishes a semantic relationship graph of cross-source data and constructs a global data association network. The data fusion rule engine unit automatically performs data merging and conflict resolution according to preset business rules and generates a unified data view.

[0064] Combination Figure 5The data service and security management module includes a unified data service interface unit, an access control unit, and a data lineage tracing unit. The unified data service interface unit provides a standardized API gateway and data service catalog and encapsulates the underlying complex data acquisition logic. The access control unit implements fine-grained data access permission management and dynamically controls the data access scope. The data lineage tracing unit records the entire lifecycle of data from collection, governance, integration to use, tracking the source, processing, and usage destination of each piece of data.

[0065] The data source access unit includes a connection management processor, an adapter generation processor, and an access scheduling processor. The connection management processor is responsible for establishing and maintaining network connections with various data sources. The adapter generation processor automatically generates or configures corresponding data adapters according to the access characteristics of different data sources. The access scheduling processor reasonably arranges data access tasks according to the priority, bandwidth limitations, and system load of the data sources.

[0066] The protocol parsing unit includes a protocol identification processor, a format conversion processor, and a verification processor. The protocol identification processor automatically identifies the protocol type used for data transmission and calls the corresponding parsing strategy. The format conversion processor converts data payloads under different protocols into a unified data format within the system. The verification processor performs integrity and legality verification on the parsed data.

[0067] The real-time acquisition unit includes a stream processor, a buffer queue manager, and a timestamp annotation processor. The stream processor uses a streaming computing architecture to continuously acquire and process real-time data. The buffer queue manager sets up a data buffer queue at the acquisition end and dynamically adjusts the acquisition rate according to the queue length. The timestamp annotation processor records accurate timestamp information for each piece of acquired data, recording the data generation time and acquisition time.

[0068] The data cleaning unit includes an anomaly detection processor, a deduplication processor, and a missing value completion processor. The anomaly detection processor identifies outliers in the data through statistical analysis and rule matching and marks or deletes them according to the configuration. The deduplication processor identifies and eliminates duplicate data records and retains the latest or most complete record version. The missing value completion processor intelligently completes missing fields in the data.

[0069] The data standardization unit includes a field mapping processor, an encoding conversion processor, and a format normalization processor. The field mapping processor maps fields from different data sources that express the same semantics but have different names to a unified standard field name. The encoding conversion processor converts the encoding systems used by different systems into a unified standard encoding. The format normalization processor performs unified processing on the data format to make all data conform to the predefined format specifications.

[0070] The data quality assessment unit includes a quality rule engine, a scoring calculation processor, and a quality report generator. The quality rule engine maintains a complete set of data quality assessment rules and supports the flexible definition and combination of rules. The scoring calculation processor scores each piece of data or dataset according to the quality rules, calculates the quality index values ​​of each dimension, and generates a comprehensive quality score. The quality report generator periodically generates data quality assessment reports and statistically analyzes the types, distribution, and trends of quality problems.

[0071] The cross-source entity recognition unit includes a feature extraction processor, a similarity calculation processor, and an entity matching processor. The feature extraction processor extracts key features for entity recognition from records from different data sources and constructs entity feature vectors. The similarity calculation processor calculates the similarity between records from different data sources and generates a similarity matrix. The entity matching processor determines which records represent the same real entity based on the similarity calculation results and performs entity merging, assigning a unique global identifier to each real entity.

[0072] The similarity calculation processor calculates the enhanced similarity between the two data points according to the following formula:

[0073] ;

[0074] Among them, S enh (i, j) represents the enhanced similarity between record i from data source A and record j from data source B, where n is the total number of feature dimensions, and w k f represents the weight coefficient of the k-th dimension feature. ik and f jk Let sim represent the feature values ​​of record i and record j in the k-th feature dimension, respectively. k Let C be the basic similarity calculation function for the k-th dimension feature. k Cross-source complementarity of the k-th dimension feature, I is the complementarity gain coefficient. i and I j These are the sets of non-empty feature dimension indices for records i and j, respectively. The feature coverage consistency gain coefficient;

[0075] The cross-source complementarity is calculated according to the following formula:

[0076] ;

[0077] Where, q Ak and q Bk These represent the average quality levels of data source A and data source B on the k-th dimension feature, respectively.

[0078] The similarity calculation processor will S enh (i, j) is the element in the i-th row and j-th column of the similarity matrix between data source A and data source B;

[0079] The basic similarity calculation function adopts some common calculation methods, and no limitation is made on the actual method used.

[0080] The entity matching processor calculates the propagation-enhanced matching score M for record i and record j according to the following formula. prop (i, j):

[0081] ;

[0082] Where H is the maximum propagation hop count, N is the propagation attenuation factor. h (i) and N h (j) represent the h-hop neighbor set of record i and record j, respectively, w ip and w jq Record the strength weights of the relationships between i and its neighboring node p, and between j and its neighboring node q, respectively.

[0083] When the propagation enhancement matching score exceeds the matching threshold, the entity matching processor will merge record i and record j into the same entity;

[0084] The semantic association mapping unit includes a relationship discovery processor, a graph construction processor, and an association path analyzer. The relationship discovery processor analyzes semantic relationships such as foreign key associations, business process associations, and temporal co-occurrence associations between data and identifies the relationship type and strength. The graph construction processor organizes the identified entities and relationships into a semantic relationship graph structure, with nodes representing data entities and edges representing relationships between entities, forming a global data association view. The association path analyzer provides graph query and traversal capabilities.

[0085] The data fusion rule engine unit includes a rule configuration manager, a conflict detection processor, and a fusion decision processor. The rule configuration manager provides functions for defining, storing, and managing fusion rules. The conflict detection processor identifies data conflicts of the same entity in different data sources during the data fusion process and marks the conflict type and conflict field. The fusion decision processor automatically adjudicates the detected conflicts according to preset fusion rules and forms a unified data view.

[0086] The fusion decision processor calculates the fusion value V of the m records merged into the same entity on the numerical attribute according to the following formula. fus Perform the calculation:

[0087] ;

[0088] Among them, V t T represents the value of the t-th record on this attribute. t Let t be the credibility weight of the data source for the t-th record. Let be the time difference of the t-th record. This is the time-related decay coefficient. The entropy sensitivity coefficient, Here, Hnorm is the median weighting coefficient, Hnorm is the normalized information entropy, and median{} is the median function.

[0089] The time difference refers to the time difference between the recorded timestamp and the current fusion time.

[0090] The normalized information entropy is calculated according to the following formula:

[0091] ;

[0092] ;

[0093] The unified data service interface unit includes an API gateway, a service registry, and a request routing processor. The API gateway serves as the unified entry point for all data service requests, providing functions such as identity authentication, request rate limiting, and protocol conversion. The service registry maintains the directory information of all available data services and supports automatic service registration and discovery. The request routing processor routes requests to the corresponding backend data services based on the type and content of the service requests.

[0094] The access control unit includes a permission policy manager, an identity authentication processor, and an authorization decision processor. The permission policy manager defines and manages data access permission policies. The identity authentication processor authenticates data access requests and confirms the true identity of the requester. The authorization decision processor makes real-time authorization decisions based on the requester's identity, the data resources accessed, the current access context, and other information, combined with the permission policies.

[0095] The data lineage tracing unit includes a lineage information collector, a lineage relationship storage unit, and a lineage tracing query processor. The lineage information collector automatically captures lineage information at each stage of data flow and constructs a complete data lifecycle trajectory. The lineage relationship storage unit stores data lineage relationships in a graph structure, where nodes represent data entities or processing procedures, and edges represent data flow and dependencies. The lineage tracing query processor provides forward tracing and reverse tracing functions. Forward tracing starts from the source data to track the use and scope of influence of the data, while reverse tracing starts from the result data to track the source and processing of the data.

[0096] The i, j, h, t and s mentioned above are ordinal numbers used to represent sequence numbers and have no actual meaning. In particular, i specifically refers to the record sequence number of data source A, and j specifically refers to the record sequence number of data source B. Data source A and data source B are temporarily specified during actual operation.

[0097] To fully verify the effectiveness and superiority of the technical solution proposed in this invention, a series of comparative experiments were designed and implemented. The experiments employed a real-world, multi-source, heterogeneous data environment within an enterprise. Data sources included five major categories: customer relationship management systems, enterprise resource planning systems, supply chain management systems, e-commerce platform interfaces, and IoT sensor devices, encompassing a total of 327 data tables and 8642 fields, with data records ranging from 100,000 to 10 million. The comparative experiments included traditional rule-based matching methods, machine learning-based similarity calculation methods, graph propagation-based matching algorithms, and the proposed solution combining enhanced similarity calculation with propagation-enhanced matching.

[0098] The experimental steps are as follows: The first step is setting up the experimental environment. The experimental system was deployed on a server configured with an Intel Xeon Gold 6248R processor, 128GB of memory, and a 2TB solid-state drive. The database used was PostgreSQL version 14.5, and the distributed computing framework was Apache Spark 3.3.0, ensuring that all comparison methods ran under the same hardware and software conditions. The second step is the data preparation stage. Raw data was extracted from five major data sources and test datasets were constructed at five scale levels: 100,000, 500,000, 1 million, 5 million, and 10 million records. Simultaneously, 5,000 pairs of entity records were manually labeled as the gold standard dataset for accuracy evaluation. The labeling work was independently completed by three domain experts and passed consistency checks. The third step is the data access and acquisition experiment, testing the protocol adaptation capabilities and real-time acquisition performance of each solution to heterogeneous data sources, recording the acquisition completion time and system resource consumption under different data scales. The fourth step is a data cleaning and standardization experiment. Anomaly detection, deduplication, missing value completion, field mapping, encoding conversion, and format normalization are performed on the collected raw data. Data quality indicators before and after cleaning, including completeness, accuracy, and consistency scores, are statistically analyzed. The fifth step is an entity recognition accuracy experiment. The entity matching performance of each scheme is evaluated using a labeled gold-standard dataset. Precision, recall, and F1 score are calculated as evaluation metrics. Precision reflects the accuracy of the matching results, recall reflects the comprehensiveness of the recognition, and the F1 score is the harmonic mean of the two to comprehensively evaluate matching performance. The sixth step is a processing efficiency experiment. The time required for each scheme to complete the entire data integration process is tested under different data scales, including the total time spent on data collection, cleaning, standardization, entity recognition, and data fusion.

[0099] Finally, the data was organized and obtained Figure 6 and Figure 7 A comparison diagram.

[0100] Example 3.

[0101] This embodiment provides a specific implementation plan for a data silo elimination and integration system in a data middleware network. The system is deployed in a cluster environment consisting of five Dell PowerEdge R750 servers. Each server is equipped with dual Intel Xeon Gold 6338 processors, 256GB DDR4-3200 ECC memory, and four 2TB NVMe SSDs forming a RAID10 array. The servers are interconnected via a Cisco Nexus 9300 10 Gigabit Ethernet switch, achieving a network bandwidth of 10Gbps. The overall system storage capacity is 40TB, which can support the real-time acquisition requirement of processing 500,000 data records per second.

[0102] The data access and acquisition module is implemented using a distributed microservice architecture, with the data source access unit deployed on three servers to form a highly available cluster. The connection management processor uses Apache Kafka 3.4.0 as the message middleware, configured with 12 partitions and 3 replication factors, capable of maintaining 500 data source connections simultaneously. The connection pool is implemented using HikariCP 5.0.1, with a minimum of 50 connections, a maximum of 200 connections, a connection timeout of 30 seconds, and an idle connection timeout of 10 minutes. The adapter generation processor is developed based on the Spring Boot 3.0 framework, employing strategy and factory patterns to implement an automated adapter generation mechanism. The system pre-configures 12 standard adapter templates, including JDBC, HTTP, MQTT, FTP, and WebSocket adapters. When a new data source is connected, the system automatically selects the appropriate adapter template and performs parameterized configuration based on the data source's metadata information, such as database type, version number, table structure, and field types. The average generation time is 3.5 seconds, representing an efficiency improvement of approximately 85% compared to manually writing adapter code. The access scheduler uses Quartz 2.3.2 as its task scheduling framework, supporting Cron expressions for configuring scheduling rules. The system categorizes data sources into three priority levels: high, medium, and low. High-priority data sources, such as transaction and order systems, are set to real-time collection intervals. Medium-priority data sources, such as customer and inventory management systems, are set to a 5-minute collection interval. Low-priority data sources, such as historical archived data, are set to a 1-hour collection interval. The scheduler also implements a dynamic adjustment mechanism based on system load. When CPU utilization exceeds 80% or memory utilization exceeds 85%, the collection frequency is automatically reduced and the execution of low-priority tasks is delayed. Normal scheduling resumes when the system load drops below 60%, ensuring stable system operation.

[0103] The protocol parsing unit is responsible for parsing various heterogeneous transmission protocols. The protocol identification processor automatically identifies the transmission protocol type by analyzing features such as the preamble byte, port number, and message format of the data packets. The system supports 18 commonly used protocols, including HTTP / HTTPS, FTP / SFTP, MQTT, AMQP, WebSocket, gRPC, and Thrift, with an accuracy rate of 99.3% and an average identification time of 12 milliseconds. The format conversion processor uses Apache Avro 1.11.1 as the intermediate serialization format to uniformly convert data payloads from different protocols to Avro format. The conversion process uses streaming processing to avoid memory overflow, with a single record conversion time of approximately 0.8 milliseconds, and can process 1.2 million records per second. JSON data is parsed using Jackson 2.14, XML data using DOM4J 2.1.3, Protobuf data using native Protobuf 3.21, and CSV data using OpenCSV 5.7. The system automatically selects the appropriate parser based on content type detection. The validation processor implements a three-layer validation mechanism. The first layer is protocol integrity validation, checking whether data packets conform to protocol specifications and whether they are truncated or corrupted, with a pass rate of 98.7%. The second layer is data validity validation, checking whether field types match, whether numerical ranges are reasonable, and whether required fields are missing. Validation rules are defined using JSON Schema, with over 500 commonly used validation rules pre-defined and users can customize and extend them through the configuration interface, achieving a pass rate of 96.2%. The third layer is business logic validation, checking whether data conforms to business constraints such as order amounts not being negative and customer ages being within a reasonable range. Business rules are configured using the Drools 8.31 rule engine, achieving a pass rate of 94.8%. For data that fails validation, the system records it in the abnormal data table and triggers an alarm notification, while also providing data repair suggestions.

[0104] The real-time acquisition unit is specifically designed to handle business data with high timeliness requirements. The stream processor uses Apache Flink 1.17.0 as its streaming computing engine, configured with 5 TaskManagers, each allocated 4 TaskSlots, for a total of 20 degrees of parallelism. Each TaskSlot is allocated 8GB of heap memory and 4GB of off-heap memory. The system uses EventTime semantics to process data, with a maximum out-of-order time of 5 seconds, a watermark generation interval of 200 milliseconds, and a sliding window type with a window size of 1 minute and a sliding step of 10 seconds. In actual operation, the system collects real-time data including order data from e-commerce platforms, transaction data from payment systems, trajectory data from logistics systems, and log data from monitoring systems. It processes an average of 1.5 billion data entries per day, with a peak processing speed of 800,000 entries per second, and end-to-end latency controlled within 500 milliseconds, meeting the needs of real-time data analysis. The buffer queue manager uses Disruptor 3.4.4 as its high-performance queue implementation, with a Ring Buffer size of 8192. It employs a multi-producer, multi-consumer model, and the BlockingWaitStrategy balances performance and CPU usage. The queue manager implements a backpressure mechanism: when the consumption rate cannot keep up with the production rate, causing queue backlog to exceed a threshold, the system automatically reduces the acquisition rate. The threshold is set at 75% of the queue capacity (6144 queues), and the rate reduction is 60% of the original rate. This mechanism effectively prevents data loss and service crashes caused by system overload. In stress testing, the system remained stable even when the input rate suddenly increased by 3 times, with a data loss rate controlled below 0.01%. The timestamp annotation processor records three pieces of time information for each piece of acquired data. The first is the data generation time, i.e., the timestamp of the data generated by the business system, extracted from the data source's time field or recorded according to the system clock. The second is the data acquisition time, i.e., the timestamp of the data received by the system, provided by the acquisition server's system clock, synchronized with a standard time server using the NTP protocol, with clock deviation controlled within 10 milliseconds. The third is the data processing time, which is the timestamp of the system completing the initial data processing. This timestamp is used to calculate processing latency and for performance analysis. The timestamp is stored in the Unix millisecond timestamp format, occupying an 8-byte long integer with millisecond-level precision, meeting the timing requirements of real-time data analysis.

[0105] The data cleaning unit implements intelligent data quality problem handling. The anomaly detection processor employs multiple detection algorithms. For numerical fields, it uses the 3-sigma principle to detect outliers, marking values ​​exceeding the mean plus or minus three standard deviations as anomalies, achieving an accuracy of 91.5%. For discrete fields, it uses frequency analysis to detect outliers, marking values ​​with a frequency below 0.1% of the total as suspicious anomalies, further analyzed in conjunction with business rules. For time-series data, it uses a sliding window to detect abrupt changes, with a window size of 10 data points. A data point deviating from the mean within the window by more than two standard deviations is marked as an abrupt change, achieving a sensitivity of 89.3%. The system also supports machine learning-based anomaly detection, using the Isolation Forest algorithm to train the anomaly detection model. The contamination parameter is set to 0.05, assuming 5% of the data is anomaly. The model, trained on a dataset containing 500,000 training samples, achieves an accuracy of 93.7%, a 2.2 percentage point improvement over rule-based methods. The anomaly detection processor also implements an adaptive threshold mechanism, recalculating statistical parameters hourly based on the latest data to adapt the detection threshold to changes in data distribution. The deduplication processor employs a two-stage deduplication algorithm combining Bloom filters and exact matching. The first stage uses a Bloom filter for rapid screening, with a filter size of 10 million bits and three hash functions, achieving a false positive rate of 0.01%. Records suspected of being duplicates after the initial Bloom filter screening proceed to the second stage for exact matching, using MD5 hash comparison to confirm duplicates. The system defines duplicate detection criteria based on business rules: for order data, the order number is used as a unique identifier; for customer data, the ID card number or mobile phone number is used; and for device data, the device MAC address is used. When a duplicate record is detected, the system retains the latest record based on its timestamp or the record with the most complete information based on its completeness. Deleted duplicate records are archived in a history table and permanently deleted after 30 days. In actual operation, the system detects approximately 2 million duplicate records daily, achieving a deduplication rate of 8.5% and a processing speed of 150,000 records per second. The missing value completion processor employs different completion strategies based on field type and business characteristics. For numeric fields, if the missing value rate is below 10%, the mean is used for imputation; if the missing value rate is between 10% and 30%, the median is used for imputation (more robust); if the missing value rate exceeds 30%, the field is marked as a low-quality field and no imputation is performed. For categorical fields, the mode is used for imputation, i.e., the most frequently occurring value is used; if multiple modes exist, one is chosen randomly. For time fields, forward imputation is used, i.e., the time value of the previous record is used; if it is the first record, the current system time is used. For text fields, short text is imputed with default values ​​such as unknown, none, or empty; long text, such as descriptive information, is left empty and not imputed.The system also implements intelligent completion based on the KNN algorithm. It selects the 5 records most similar to the current record and fills the gaps using the average or mode of the corresponding fields of these 5 records. The similarity calculation uses Euclidean distance, and the completion accuracy reaches 87.2%, which is 12.6 percentage points higher than the simple statistical method.

[0106] The data standardization unit ensures semantic and format consistency across different data sources. The field mapping processor maintains a data dictionary containing over 2000 standard fields, organized in a three-tiered structure. The first tier comprises 12 business domains, such as customer, order, product, and financial domains. The second tier includes 80 entity types, such as customer, order, and product entities. The third tier contains over 2000 field attributes, such as customer name, customer age, order amount, and product price. Each standard field defines attributes including a standard name, data type, value range, business meaning, and example value. The field mapping processor establishes mapping relationships in three ways: first, fuzzy matching based on field names, using an edit distance algorithm to calculate similarity with a similarity threshold of 0.85, achieving a matching success rate of 72.3%; second, pattern recognition based on field content, analyzing data distribution characteristics such as value range, data format, and statistical properties to infer field semantics, achieving an accuracy rate of 81.7%. The third method is supervised learning based on manual annotation. The system has accumulated an annotated dataset containing 100,000 mapping pairs and trained a BERT-based semantic matching model, achieving an accuracy of 94.5% on the test set. The combined use of these three methods results in an automatic mapping coverage rate of 89.6%. For fields that cannot be automatically mapped, the system prompts for manual confirmation, and the manually confirmed mappings are added to the training dataset for continuous model optimization. The encoding conversion processor handles the conversion between different encoding systems. The system pre-configures 25 commonly used encoding standards, including the national standard industry classification GB / T 4754-2017, the international classification of diseases ICD-10, commodity codes HS Code, and regional codes GB / T 2260-2007. For administrative division codes, the system maintains a five-level encoding mapping table from the provincial to the street level, containing the encoding correspondences of 34 provincial-level administrative regions, 333 prefecture-level administrative regions, 2844 county-level administrative regions, and over 40,000 townships and streets nationwide, supporting mutual conversion between 6-digit, 9-digit, and 12-digit encoding formats. For product category coding, the system maintains a mapping table between internal enterprise codes and national standard codes, covering coding mappings for 15 major categories, 128 medium categories, and 561 minor categories. The coding conversion process is accelerated using memory caching, caching 100,000 commonly used coding mappings in Redis 7.0, achieving a cache hit rate of 96.5%. The average conversion time per transaction has been reduced from 15 milliseconds to 1.2 milliseconds, a performance improvement of over 12 times. The format normalization processor unifies data formats, converting date and time formats to the ISO 8601 standard format, yyyy-MM-dd HH:mm:ss. The original data contained 18 different formats, including yyyy / MM / dd, dd-MM-yyyy, and yyyyMMddHHmmss. The system identifies and converts these formats using regular expressions, achieving a conversion success rate of 99.1%.Telephone numbers are uniformly converted to the E.164 international standard format, i.e., +86-xxx-xxxx-xxxx. The original data contained seven different formats, including those with parentheses, hyphens, and no separators. The system automatically added country codes and separators. Amounts are uniformly formatted to retain two decimal places with a thousands separator. Currency symbols uniformly adopt the ISO 4217 standard three-letter codes such as CNY, USD, and EUR. Addresses are uniformly split into five fields: province, city, district, street, and detailed address. Natural language processing technology is used for address parsing, achieving an accuracy rate of 93.8%.

[0107] The data quality assessment unit performs a comprehensive evaluation of the governed data. The quality rule engine, based on the Drools rule engine, maintains a rule base containing over 800 quality rules, categorized into six main types: integrity rules, accuracy rules, consistency rules, timeliness rules, uniqueness rules, and compliance rules. Integrity rules check whether required fields are empty. The system defines a list of required fields for each entity; for example, for the customer entity, customer name, contact number, and region are required fields, while for the order entity, order number, customer number, order amount, and order time are required fields. Accuracy rules check whether field values ​​conform to business logic, such as age must be between 0 and 150, order amount must be greater than 0, and date cannot be later than the current time. The system defines value ranges and constraints for over 2000 fields. Consistency rules check data consistency across fields and tables, such as the total order amount should equal the sum of the order details, and the customer's age should match the calculated date of birth. Timeliness rules check the update time of data, requiring that the update time of key data does not exceed a threshold. For example, inventory data requires real-time updates with a delay of no more than 5 minutes, and customer information requires daily updates. Uniqueness rules check whether primary key and unique constraint fields have duplicate values. Compliance rules check whether the data complies with regulatory requirements, such as personal information anonymization and sensitive data encryption. The scoring processor assigns a quality score to each data entry and dataset. The quality score of a single data entry is calculated using a weighted average method. The weights of the six rule categories are: completeness 20%, accuracy 30%, consistency 25%, timeliness 10%, uniqueness 10%, and compliance 5%. Each rule is scored as 1 point for passing and 0 points for failing. The score for a rule category is the average of the scores of all rules in that category. The final quality score is the weighted sum of all scores multiplied by 100 to obtain a percentage score. The quality score of the dataset is the average of the quality scores of all records in the dataset. The system also calculates 15 quality indicators across dimensions, including completeness rate, accuracy rate, consistency rate, timeliness rate, uniqueness rate, compliance rate, null value rate, outlier rate, duplication rate, and standardization rate. The quality report generator automatically generates a data quality report for the previous day at 2 AM daily. The report, in HTML format, includes four parts: Quality Overview, Quality Trends, Issue Distribution, and Improvement Suggestions. It is sent via email to data administrators and relevant business managers. The Quality Overview displays key indicators such as overall quality score, scores for each dimension, total data volume, and the amount of problematic data. The Quality Trends section shows a line graph illustrating the quality score change over the past 30 days, helping managers understand the improvement or deterioration of data quality. The Issue Distribution section displays a pie chart showing the percentage of different types of quality issues and a bar chart comparing the quality scores of different data sources, helping to pinpoint the source of quality issues. Improvement suggestions are automatically generated based on statistical analysis of quality issues; for example, if the missing value rate of a certain data source remains consistently high, suggestions include optimizing the data collection program or contacting the data provider for improvements.

[0108] The cross-source entity recognition unit is the core component of data fusion. The feature extraction processor extracts features for entity matching from data records. Feature types include identifier features such as unique fields like customer ID, national ID number, and mobile phone number; attribute features such as descriptive fields like customer name, address, and email; relational features such as fields reflecting relationships between entities like affiliated organization and related orders; and statistical features such as transaction count and cumulative amount. For textual features, the system uses the TF-IDF algorithm to extract keywords and construct word vectors, with a word vector dimension of 128. For numerical features, MinMax normalization is used to map them to the 0-1 range. For categorical features, One-Hot encoding is used to convert them into binary vectors. Missing feature dimensions are filled with default values ​​of 0. Finally, each record is constructed as a feature vector containing 50 feature dimensions. The similarity calculation processor calculates the similarity between records from different data sources, introducing cross-source complementarity and feature coverage consistency gains on top of basic similarity calculation. Cross-source complementarity measures the degree of complementarity between two data sources in terms of feature quality. The system maintains the average quality score of each data source across 50 feature dimensions. The quality score is calculated based on indicators such as the feature's completeness and accuracy, and ranges from 0 to 1. A higher quality data source A in a certain feature dimension and a lower quality data source B in that dimension indicates higher complementarity, and vice versa. Feature coverage consistency gain measures the consistency between two records in terms of feature completeness. Positive gain is contributed when both records have non-null values ​​in a certain feature dimension; no gain is contributed when only one record has a value; and negative gain is contributed when both are null. By introducing these two gain factors, the system can more accurately assess the similarity between cross-source records. In experiments, the recognition accuracy was improved by 11.3 percentage points compared to the traditional cosine similarity method. The entity matching processor performs entity matching based on the similarity matrix. When the similarity exceeds the matching threshold of 0.85, the entity is identified as the same entity. The system also implements a propagation-enhanced matching mechanism, utilizing the correlation between data for global optimization. For example, when record A is similar to record C and record B is similar to record C, even if the direct similarity between record A and record B is not high, they are likely to be the same entity. The system incorporates this indirect relationship into the matching judgment through the graph propagation algorithm, with the propagation hop count set to 3 hops and the propagation attenuation factor set to 0.7, that is, the weight of one propagation hop is attenuated to 70%.

[0109] The semantic association mapping unit constructs a semantic relationship network across source data. The relationship discovery processor identifies semantic connections by analyzing the relationships between data, including foreign key relationships (primary and foreign key constraints between database tables), business process relationships (reflecting upstream and downstream relationships in a business process, such as the relationship between an order and a delivery note, or a delivery note and a receiving note), temporal co-occurrence relationships (records appearing within similar time windows may be related, such as multiple browsing behaviors of the same customer within 10 minutes), spatial co-occurrence relationships (records appearing in similar geographical locations may be related, such as multiple stores in the same business district), and attribute similarity relationships (records with similar attributes may be related, such as products at similar price points, or customers of similar ages). The system uses the Apriori association rule mining algorithm to discover frequent itemsets, with a minimum support set to 0.01 (the association appears more than 1% of the total number of itemsets) and a minimum confidence set to 0.6 (the association's confidence exceeds 60%). In the correlation analysis of customer and order data, the system discovered 218 strong correlation rules between customer age groups and product categories, such as 20-30 year old customers tending to buy digital products (confidence 0.73) and 30-40 year old customers tending to buy maternity and baby products (confidence 0.68). The graph construction processor organizes the identified entities and relationships into a graph structure for storage, using Neo4j 5.5.0 graph database as the storage engine. The graph database is deployed on a dedicated server with 128GB of memory and 4TB of SSD storage. Nodes in the graph represent data entities, and node attributes include entity identification information, basic attributes, quality scores, etc. Node types include 15 entity types such as customer nodes, order nodes, product nodes, store nodes, and device nodes, with a total of 12 million nodes. Edges in the graph represent relationships between entities, and edge attributes include relationship type, relationship strength, establishment time, etc. Relationship types include 25 relationship types such as purchase relationships, browsing relationships, favorite relationships, rating relationships, and recommendation relationships, with a total of 85 million edges. To optimize graph query performance, the system indexed frequently accessed node attributes, created unique constraint indexes for key identifier fields such as customer ID, order number, and product code, and established a range index for the timestamp field. The index coverage reached 82%, reducing the average response time for single-hop queries from 120 milliseconds to 15 milliseconds and for multi-hop queries from 2.3 seconds to 0.8 seconds. The association path analyzer provides graph traversal and path query capabilities, supporting various graph algorithms such as shortest path query, K-degree association query, common neighbor query, and community discovery. In customer relationship network analysis, by calculating relationships such as jointly purchased products and jointly followed stores among customers, customer groups with similar consumption preferences were identified. The Label Propagation algorithm was used for community discovery, identifying 857 customer communities with an average community size of 1823 members, an internal community connection density of 0.42, and an inter-community connection density of 0.08, providing data support for precision marketing.

[0110] The data fusion rule engine unit enables intelligent data fusion decision-making. The rule configuration manager provides a visual rule configuration interface, allowing users to configure fusion rules via drag-and-drop without writing code. Fusion rules are divided into two categories: attribute-level rules and record-level rules. Attribute-level rules define fusion strategies for individual attributes, such as using the latest value, most complete value, average value, weighted average, or voting. Record-level rules define fusion strategies for the entire record, such as using the most reliable data source or integrating multi-source information. The system pre-sets 35 commonly used fusion rule templates, which users can select and set parameters for quick configuration. The rule configuration interface also provides a rule testing function, allowing users to input test data to preview fusion results and ensure the rules meet expectations before applying them to the production environment. All fusion rules are stored in a MySQL database, using JSON format to represent rule conditions and actions, and supporting rule version management and rollback. The conflict detection processor identifies conflicting data before data fusion. Conflict types include value conflicts (the same attribute has different values ​​in different data sources), format conflicts (the same attribute uses different data formats), semantic conflicts (the same concept is expressed differently in different data sources), and integrity conflicts (one data source contains more complete information than others). The system employs multi-dimensional analysis to identify conflicts. For numerical attributes, a conflict is identified when the deviation between values ​​from different data sources exceeds a threshold. This threshold is adaptively set based on the attribute's coefficient of variation; attributes with high coefficients of variation have higher thresholds (e.g., 30%), while those with low coefficients have lower thresholds (e.g., 5%). For categorical attributes, inconsistencies between values ​​from different data sources are directly identified as conflicts. For time-related attributes, a conflict is identified when the time difference between different data sources exceeds a reasonable range. This reasonable range is set based on business characteristics, such as allowing a deviation of 0 days for a customer's birthday or 60 seconds for an order time. In actual operation, the system detects approximately 850,000 data conflicts daily, with value conflicts accounting for 62%, format conflicts for 23%, semantic conflicts for 11%, and integrity conflicts for 4%. The fusion decision processor automatically adjudicates detected conflicts according to preset fusion rules. For numerical attributes, the system comprehensively considers multiple factors such as data source reliability, data timeliness, and data consistency for weighted fusion. Data source credibility weights are determined based on historical data quality scores. Data sources with higher quality scores have greater weights. The system maintains the quality score history of each data source for the past 90 days and uses an exponentially weighted moving average to calculate the current credibility. Recent data has a higher weight, and the decay coefficient is set to 0.95. Data timeliness is reflected by a time decay factor. The decay factor decreases by 5% for every additional day of time difference. For data with high real-time requirements, such as inventory data and price data, the timeliness decay factor is set to 0.15, meaning a 15% decay per day. For stable data, such as basic customer information, the timeliness decay factor is set to 0.02, meaning a 2% decay per day.Data consistency is measured using normalized information entropy. When multi-source data is highly consistent, the entropy value is low, and the fusion strategy tends towards a weighted average. When multi-source data shows significant discrepancies, the entropy value is high, and the fusion strategy incorporates robust median estimation to avoid the influence of outliers. The median weight coefficient is adaptively adjusted based on the entropy value. When the entropy value is below 0.3, the median weight is 0; when the entropy value is between 0.3 and 0.7, the median weight increases linearly; and when the entropy value is above 0.7, the median weight reaches 0.4. In a customer age fusion experiment, the age data from the CRM system had a reliability of 0.92 and a time difference of 15 days; the age data from the ERP system had a reliability of 0.85 and a time difference of 3 days; and the age data from the e-commerce platform had a reliability of 0.88 and a time difference of 1 day. The age values ​​from the three data sources were 32, 33, and 32 years old, respectively. The calculated normalized information entropy was 0.45, and the final fusion result was 32.3 years old, which is more scientific and reasonable than the simple average of 32.3 years old or the latest value of 32 years old. For categorical attributes, the system uses a voting mechanism for fusion, with each data source casting one vote. In cases of a tie, the value from the data source with the highest credibility is selected, achieving an accuracy rate of 96.5% in product category fusion experiments. For textual attributes, the system adopts the principle of maximum completeness, selecting the text with the most information as the fusion result. Information content is measured by text length and the number of keywords. In product description fusion experiments, the completeness of the fused text improved by an average of 37.2% compared to a single data source.

[0111] The unified data service interface unit provides standardized data access services for upper-layer applications. The API gateway is implemented using Spring Cloud Gateway 4.0.0, deployed on two servers to form a highly available architecture. Load balancing is performed using Nginx 1.24.0, employing a weighted round-robin strategy with weights dynamically adjusted based on server performance. The gateway implements authentication using a JWT token mechanism. The token's validity period is set to 2 hours, and it contains user identity, role, and permissions information. RSA-2048 algorithm signing ensures the token cannot be tampered with. The gateway also implements request rate limiting, using a token bucket algorithm to control the request rate. The QPS is limited to 100 for ordinary users, 500 for advanced users, and 2000 for VIP users. Requests exceeding these limits return a 429 status code. The gateway also provides protocol conversion functionality, supporting multiple protocols such as HTTP, gRPC, and WebSocket. Backend services uniformly use the internal RPC protocol for communication, and the gateway handles protocol conversion, allowing the frontend to access the service using any protocol. The service registry is implemented using Nacos 2.2.0 and maintains a service directory containing 85 data services. Each service includes information such as service name, version number, interface description, input parameters, output format, and call examples. The service directory provides a web interface for developers to access and also offers online testing functionality. Developers can directly construct request parameters on the page to call services and view the return results, accelerating the application development process. The service registry also implements service health checks, sending heartbeat requests to service instances every 30 seconds. If three consecutive heartbeats fail, the service instance is marked as unhealthy and removed from the service list. It is automatically re-added when the service instance recovers. This health check mechanism ensures high service availability; when a service instance fails, requests are automatically routed to other healthy instances, without user awareness. The request routing processor routes requests to the appropriate backend services based on the type and content of the service request. Routing strategies support various matching methods, including URL path, request headers, and request parameters. For query requests such as customer information queries and order list queries, they are routed to the read service cluster, which deploys five service instances using round-robin load balancing. For write requests such as data updates and deletions, they are routed to the write service cluster. The write service cluster deploys three service instances and uses consistent hashing load balancing to ensure that write requests for the same entity are routed to the same instance. The routing processor also implements intelligent routing functionality, selecting the optimal service instance based on factors such as the amount of data requested, complexity, and response time requirements, prioritizing instances with lower load and shorter response times.

[0112] The access control unit enables fine-grained data access management. The permission policy manager employs the RBAC (Role-Based Access Control) model, pre-defining 15 roles, including data administrator, data analyst, business personnel, developers, and external partners, each with different data access permissions. Data administrators have the highest privileges, accessing all data and performing CRUD operations. Data analysts can access the full, anonymized data for analysis but cannot modify it. Business personnel can only access business data relevant to their department (e.g., sales department can only access sales data, purchasing department can only access purchasing data). Developers can access test environment data but not sensitive production environment data. External partners can only access authorized specific datasets, and their access is strictly audited. Permission granularity is refined to three levels: table, row, and column. Table-level permissions control which tables a user can access; row-level permissions control which records within a table a user can access (e.g., sales personnel can only access customer records in their assigned region); and column-level permissions control which fields within a record a user can access (e.g., ordinary employees cannot access sensitive fields such as customer ID numbers and bank account numbers). The authorization decision processor makes real-time authorization decisions based on the requester's identity, the accessed data resource, and the current access context. The authorization decision-making adopts the ABAC attribute-based access control model, comprehensively considering multiple dimensions such as user attributes (role, department, level), resource attributes (data category, sensitivity level, department), and environmental attributes (access time, location, device). The authorization decision engine adopts an architecture that separates the policy decision point (PDP) and the policy execution point (PEP). The PDP is responsible for making authorization decisions, and the PEP is responsible for executing the decision results.

[0113] The data lineage tracing unit enables traceable management of the entire data lifecycle. The lineage information collector automatically captures lineage information at each stage of data flow. Collection points include: data collection stage (recording data source, collection time, and collection method); data cleaning stage (recording cleaning rules, processing results, and deleted or modified records); data standardization stage (recording mapping rules, transformation process, and field changes); data fusion stage (recording fusion rules, participating source records, and fusion results); and data service stage (recording data users, usage time, and usage purpose). Lineage information collection is implemented using Aspect-Oriented Programming (AOP) technology, automatically intercepting and recording lineage information before and after the execution of critical processing methods without intruding on business code. Lineage information includes fields such as data entity identifier, operation type, operation time, operator, input data, output data, processing rules, and processing parameters, stored in JSON format. The amount of lineage information is very large; the system generates approximately 50 million lineage records daily. To avoid impacting business system performance, lineage information is written asynchronously, first to a Kafka message queue, and then batch-written to the lineage database by a dedicated consumer program. The lineage relationship storage uses the Neo4j graph database to store lineage relationships. Nodes in the lineage graph represent data entities or processing procedures, and node types include source data nodes, intermediate data nodes, result data nodes, acquisition and processing nodes, cleaning and processing nodes, standardization and processing nodes, and fusion and processing nodes. Edges represent data flow and dependencies, and edge types include those originating from a relation, cleaned into a relation, transformed into a relation, merged into a relation, and used in a relation. Edge attributes record information such as flow time, processing rules, and data changes. The lineage graph storage adopts a layered design. The first layer is the summary layer, storing coarse-grained lineage relationships, such as table-level lineages. The second layer is the detailed layer, storing fine-grained lineage relationships, such as field-level lineages. The third layer is the instance layer, storing the lineage of specific data records. This layered design balances query performance and storage cost; high-level lineage queries can be completed directly at the summary layer for fast response, while detailed lineage tracing requires associating multiple layers of data. The lineage data employs a hot-and-cold separation storage strategy. Hot data from the last 30 days is stored on SSDs, warm data from 30 days to one year is stored on HDDs, and cold data older than one year is archived in Object Storage Service (OSS). The lineage tracing query processor provides forward and reverse tracing capabilities. Forward tracing starts from the source data to track its usage and impact. For example, if a data source has a quality issue, forward tracing can quickly locate all affected downstream data and applications. Reverse tracing starts from the result data to track its source and processing. For example, if an anomaly is found in a statistical report, reverse tracing can trace back to the original data source, intermediate processing steps, and the processing rules and parameters at each step, quickly pinpointing the root cause of the problem. The lineage tracing query supports various filtering conditions, such as by time range, by data source, and by processing type. Query results are presented graphically, intuitively displaying the data flow path and dependencies.The system provides a lineage impact analysis function, which allows for pre-analysis of the impact scope and assessment of change risks when planning to change a data source or processing rule. The system also offers lineage statistical analysis, tracking the usage frequency of each data source, the number of references to each data table, and the number of executions of each processing step, providing a basis for data governance and system optimization decisions.

[0114] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. A system for resolving and integrating data silos in a data middleware network, characterized in that, It includes a data access and acquisition module, a data standardization and governance module, a data fusion and association module, and a data security management and control module. The data access and acquisition module opens up data input channels from different sources and achieves efficient aggregation of diverse data. The data standardization and governance module is used to clean, standardize, and control the quality of the accessed data and eliminate semantic and structural differences. The data fusion and association module achieves deep integration of cross-source data through identification, association, and fusion technologies. The data security management and control module is used to provide secure and controllable open services for the integrated data. The data access and acquisition module includes a data source access unit, a protocol parsing unit, and a real-time acquisition unit. The data source access unit is responsible for establishing connections with business systems, third-party platforms, sensor devices, etc., to ensure that various data sources can be smoothly accessed by the system. The protocol parsing unit performs unified parsing of different transmission protocols used by heterogeneous data sources to eliminate differences at the protocol level. The real-time acquisition unit performs real-time streaming acquisition of business data that requires rapid response, such as transaction data and log data. The data standardization and governance module includes a data cleaning unit, a data standardization unit, and a data quality assessment unit. The data cleaning unit detects and processes data quality issues, the data standardization unit maps data content from different systems to a unified data standard model, and the data quality assessment unit establishes a quality indicator system and automatically assesses and labels the governed data. The data fusion and association module includes a cross-source entity recognition unit, a semantic association mapping unit, and a data fusion rule engine unit. The cross-source entity recognition unit identifies records representing the same real entity scattered across different data sources and provides a basis for data association. The semantic association mapping unit establishes a semantic relationship graph of cross-source data and constructs a global data association network. The data fusion rule engine unit automatically performs data merging and conflict resolution according to preset business rules and generates a unified data view. The data service and security management module includes a unified data service interface unit, an access control unit, and a data lineage tracing unit. The unified data service interface unit provides a standardized API gateway and data service catalog and encapsulates the underlying complex data acquisition logic. The access control unit implements fine-grained data access permission management and dynamically controls the data access scope. The data lineage tracing unit records the entire lifecycle of data from collection, governance, integration to use, tracking the source, processing, and usage destination of each piece of data.

2. The data silo elimination and integration system in a data middleware network as described in claim 1, characterized in that, The cross-source entity recognition unit includes a feature extraction processor, a similarity calculation processor, and an entity matching processor. The feature extraction processor extracts key features for entity recognition from records from different data sources and constructs entity feature vectors. The similarity calculation processor calculates the similarity between records from different data sources and generates a similarity matrix. The entity matching processor determines which records represent the same real entity based on the similarity calculation results and performs entity merging, assigning a unique global identifier to each real entity.

3. The data silo elimination and integration system in a data middleware network as described in claim 2, characterized in that, The similarity calculation processor calculates the enhanced similarity between the two data points according to the following formula: ； Among them, S enh (i, j) represents the enhanced similarity between record i from data source A and record j from data source B, where n is the total number of feature dimensions, and w k f represents the weight coefficient of the k-th dimension feature. ik and f jk Let sim represent the feature values ​​of record i and record j in the k-th feature dimension, respectively. k Let C be the basic similarity calculation function for the k-th dimension feature. k Cross-source complementarity of the k-th dimension feature, I is the complementarity gain coefficient. i and I j These are the sets of non-empty feature dimension indices for records i and j, respectively. This is the feature coverage consistency gain coefficient.

4. The data silo elimination and integration system in a data middleware network as described in claim 3, characterized in that, The entity matching processor calculates the propagation-enhanced matching score M for record i and record j according to the following formula. prop (i, j): ； Where H is the maximum propagation hop count, N is the propagation attenuation factor. h (i) and N h (j) represent the h-hop neighbor set of record i and record j, respectively, w ip and w jq Record the strength weights of the relationships between i and its neighboring node p, and between j and its neighboring node q, respectively. When the propagation enhancement matching score exceeds the matching threshold, the entity matching processor will merge record i and record j into the same entity.

5. The data silo elimination and integration system in a data middleware network as described in claim 4, characterized in that, The data fusion rule engine unit includes a rule configuration manager, a conflict detection processor, and a fusion decision processor. The rule configuration manager provides functions for defining, storing, and managing fusion rules. The conflict detection processor identifies data conflicts of the same entity in different data sources during the data fusion process and marks the conflict type and conflict field. The fusion decision processor automatically adjudicates the detected conflicts according to preset fusion rules and forms a unified data view. The fusion decision processor calculates the fusion value V of the m records merged into the same entity on the numerical attribute according to the following formula. fus Perform the calculation: ； Among them, V t T represents the value of the t-th record on this attribute. t Let t be the credibility weight of the data source for the t-th record. Let be the time difference of the t-th record. This is the time-related decay coefficient. The entropy sensitivity coefficient, Here, Hnorm is the median weighting coefficient, Hnorm is the normalized information entropy, and median{} is the median function.

Images