Heterogeneous data fusion method and device based on bridging technology, equipment and medium

Abstract

The application provides a heterogeneous data fusion method and device based on a bridging technology, equipment and a medium, and belongs to the technical field of data processing. The method comprises the following steps: acquiring source data, performing data extraction on the source data to obtain metadata in the source data; for each source field in the metadata, the source field is matched with a standard field in a preset standard data model to obtain a standard field matched with the source field; a mapping rule is determined based on the matching relationship between the source field and the standard field, the source data is mapped to the standard data model based on the mapping rule, and converted source data is obtained; and the converted source data is used for fusion of multi-source heterogeneous data. The heterogeneous data fusion method and device based on the bridging technology, the equipment and the medium provided by the application can realize efficient and reliable data fusion.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and more specifically, relates to a heterogeneous data fusion method, apparatus, device, and medium based on bridging technology. Background Technology

[0002] With the rapid development of IoT, cloud computing, and AI technologies, data is experiencing explosive growth, with increasingly diverse sources and more complex forms. In numerous fields such as smart cities, smart energy, industrial IoT, intelligent manufacturing, financial risk control, medical diagnosis, and national defense, decision-making often relies on the comprehensive analysis of "multi-source heterogeneous data" from different systems, formats, and semantics. These data sources may include relational databases, NoSQL databases, time-series data generated by sensor networks, text files, images, videos, and social media streams.

[0003] Different data sources employ different storage systems, communication protocols, and data models (such as relational models, document models, key-value models, etc.), making physical interconnection extremely difficult. Even data that can be physically connected may have completely different schemas, naming conventions, data units, and encoding methods for describing the same thing, creating semantic gaps of "homonyms" or "hypotheticals," which is the core obstacle to data fusion.

[0004] To address these challenges, academia and industry have proposed various data fusion frameworks. For example, the mediator / wrapper-based model performs initial data format conversion by developing a dedicated wrapper / adapter for each data source, followed by a centralized mediator handling query decomposition and result merging. However, traditional mediator models remain insufficient when dealing with highly dynamic and complex semantic heterogeneity. Therefore, a novel, more flexible, and intelligent data fusion method is urgently needed to achieve efficient and reliable data fusion. Summary of the Invention

[0005] The purpose of this application is to provide a heterogeneous data fusion method, apparatus, device, and medium based on bridging technology to achieve efficient and reliable data fusion.

[0006] A first aspect of this application provides a heterogeneous data fusion method based on bridging technology, comprising: Obtain source data, extract data from the source data, and obtain metadata from the source data; For each source field in the metadata, the source field is matched with a standard field in a preset standardized data model to obtain a standard field that matches the source field. Based on the matching relationship between the source field and the standard field, a mapping rule is determined, and the source data is mapped to the standardized data model based on the mapping rule to obtain the transformed source data. The transformed source data is used to fuse multi-source heterogeneous data.

[0007] A second aspect of this application provides a heterogeneous data fusion device based on bridging technology, comprising: The data acquisition module is used to acquire source data, extract data from the source data, and obtain metadata from the source data. The field matching module is used to match each source field in the metadata with a standard field in a preset standardized data model to obtain a standard field that matches the source field. The data mapping module is used to determine mapping rules based on the matching relationship between the source field and the standard field, and to map the source data to the standardized data model based on the mapping rules to obtain the transformed source data. The data fusion module is used to fuse multi-source heterogeneous data based on the transformed source data.

[0008] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the heterogeneous data fusion method based on bridging technology described above.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the heterogeneous data fusion method based on bridging technology described above.

[0010] The beneficial effects of the heterogeneous data fusion method, apparatus, device, and medium based on bridging technology provided in this application are as follows: This application's embodiments simplify the complex "many-to-many" data source connection problem into multiple "one-to-one" source-to-bridge mapping problems by introducing a standardized data model as a data bridge, significantly reducing the complexity of system integration and development and maintenance costs. Specifically, during the mapping process from source data to the standardized data model, semantic alignment between the source data and the standardized data model is achieved through field matching, ensuring consistency of data at the business meaning level and improving the accuracy and usability of the fused data. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a heterogeneous data fusion method based on bridging technology provided in an embodiment of this application; Figure 2 A flowchart illustrating a heterogeneous data fusion method based on bridging technology, provided in another embodiment of this application; Figure 3 An architecture diagram of a heterogeneous data fusion system based on bridging technology provided in an embodiment of this application; Figure 4 This is a structural block diagram of a heterogeneous data fusion device based on bridging technology provided in an embodiment of this application; Figure 5 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0015] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0017] Please refer to Figure 1 , Figure 1This is a flowchart illustrating a heterogeneous data fusion method based on bridging technology according to an embodiment of this application. The heterogeneous data fusion method based on bridging technology provided in this embodiment can be executed by an electronic device, and the method may include: S101: Obtain source data, extract data from the source data, and obtain metadata from the source data.

[0018] In this embodiment, the source data may include various heterogeneous data sources that need to be merged, such as internal relational databases (MySQL, Oracle), business log files, sensor data streams (MQTT) on IoT platforms, public Web APIs, and unstructured document (PDF, Word) and image data repositories.

[0019] Data extraction is performed on the acquired source data. For example, the scanning module of the metadata management platform can be used to extract data describing the core characteristics of the source data, thus obtaining the metadata in the source data. Metadata can include the name of the source field, data type (such as int, string, datetime), field length, value range, NOT NULL constraints, data organization (such as primary / foreign keys in database tables, hierarchical structure of JSON), and data relationships, etc.

[0020] S102: For each source field in the metadata, match the source field with the standard fields in the preset standardized data model to obtain the standard fields that match the source field.

[0021] In this embodiment, based on the needs of the target business domain, a standardized data model, independent of any specific data source or application system, can be designed that is relevant to the specific business domain. This model defines the core business entities, attributes, and relationships between them within the domain. As a unified target format for data fusion, it acts as a "common language" or "unified target," and all data from different source systems needs to be mapped to this model during integration.

[0022] Taking the power sector as an example, based on the State Grid Corporation of China's Unified Enterprise Information Model (SG-CIM), the data model design was completed, including data subject domains, core entities, relationship descriptions, and enterprise-level common standard codes across the entire business scope of the State Grid Corporation. The subject domains include ten categories: personnel, finance, materials, projects, power grid, assets, customers, market, safety, and comprehensive. Core entities include power plants, transformers, lines, and towers. Common standard codes include voltage levels, asset classifications, defect types, and plan types. Attributes include voltage levels, asset types, affiliated units, and dispatching units. Relationships include tower-line, line-power plant, and line-transformer relationships.

[0023] Based on this, the source field can be matched with the standard fields in the predefined canonical data model to obtain the standard field that matches the source field. For example, the "CUST_ID" field in the source data table can be mapped to the "Customer Number" field of the "Customer" entity in the canonical data model.

[0024] S103: Determine the mapping rules based on the matching relationship between the source fields and the standard fields, and map the source data to the standardized data model based on the mapping rules to obtain the transformed source data.

[0025] In this embodiment, mapping rules between the source data schema and the standardized data model can be established based on the matching relationship between the source fields and the standard fields. According to the mapping rules, corresponding data conversion logic, including data type, formatting, and standard encoding, is generated for each data source. This conversion logic includes not only field mapping but also data type conversion (e.g., string to date), data format standardization (e.g., "male / female" to "1 / 0"), and data cleaning (e.g., removing spaces and handling missing values).

[0026] Taking the multi-source heterogeneous data fusion application of a power dispatch center as an example, the power dispatch center needs to integrate the following multiple heterogeneous data sources, specifically: (1) Production Management System (PMS): Oracle relational database, containing equipment ledger information such as substations, lines, and transformers; (2) Power Grid Dispatch Technical Support System (OMS): Stores data and exchanges data through the DM database or e-file; (3) Defect Management System: MySQL database, storing device defect records; (4) GIS (Geographic Information System): Spatial location data provided via REST API; The aforementioned systems were developed by different vendors, have different data formats, and suffer from serious technical and semantic heterogeneity issues. Please refer to [reference needed]. Figure 2 The steps for multi-source heterogeneous data fusion in this power dispatch center include: Step 1: Construct a standardized data model:

[0027] First, in accordance with the requirements of the State Grid Unified Data Model (SG-CIM) specification, this embodiment defines the data model DWD_GRID_SGSUBSTATION for the power equipment subject domain "substation". Detailed information about the data model is shown in Table 1 above.

[0028] Step 2: Data Access and Adaptation Define different database or data file adapters based on the database type and data exchange method of the PMS and OMS systems: (1) Develop a relational database adapter for the PMS system. The original table structure of the PMS system is shown in Table 2 below:

[0029] Develop a relational database adapter for the PMS system to establish Oracle database connections, execute SQL queries, and extract metadata (such as table structure and field definitions). An example adapter configuration is shown below:

[0030] (2) Develop an efile file adapter for the OMS system. The OMS system CIM / E file data format is as follows:

[0031] Develop a DM database and CIM / E file adapter for the OMS system to read and parse CIM / E file data, including metadata (such as table structure and field definitions) and data, as shown below:

[0032] Step 3: Application of the semantic mapping engine: Match the source fields of each source data with the standard fields in the standardized data model. For example, match edrl in the PMS system to TRANSFORCAPACITY in the standardized data model.

[0033] Step 4: Pattern Mapping Configuration: Based on the semantic mapping results, the complete mapping rules from the PMS system to the canonical model are configured as shown in Table 3 below:

[0034] The State Grid Corporation's enterprise-level relevant data reference data standard classification names and standard classification codes are shown in Table 4 below. All subsequent fields need to be converted according to the codes.

[0035] Table 4 - Standard Classification Names and Standard Classification Codes

[0036] Standard codes are used for mapping data to standard codes. Detailed standard codes for voltage levels are shown in Table 5 below: Table 5 - Detailed Standard Codes for Voltage Levels

[0037] Step 5: Data Transformation (1) Data extraction: Raw data from the substation was extracted from the PMS system. Sample data is shown in Table 6 below.

[0038] Raw data was extracted from the OMS system, and sample data is shown in Table 7 below: Table 7 - Sample Data for OMS System Substations

[0039] (2) Data bridging and conversion: Upon entering the conversion layer, the conversion logic is executed. The specific conversion rules and validation fields are shown in Table 8 below: Table 8 - Conversion Rules

[0040] The converted, standardized data:

[0041] (3) Data quality verification: Execute the preset verification rules. Examples of the verification rules are shown in Table 9 below:

[0042] S104: Fusion of multi-source heterogeneous data based on the transformed source data.

[0043] In this embodiment, data that has passed the data quality verification can be loaded into a multi-source data fusion data warehouse for data fusion and to support the subsequent implementation of intelligent scheduling and auxiliary decision-making functions.

[0044] As can be seen from the above, this embodiment simplifies the complex "many-to-many" data source connection problem into multiple "one-to-one" source-to-bridge mapping problems by introducing a standardized data model as a data bridge, greatly reducing the complexity of system integration and development and maintenance costs. In particular, during the mapping process from source data to the standardized data model, semantic alignment between the source data and the standardized data model is achieved through field matching, ensuring consistency of data at the business meaning level and improving the accuracy and usability of the fused data.

[0045] In one embodiment of this application, for each source field in the metadata, the source field is matched with a standard field in a preset canonical data model, including: Calculate the first similarity between the source field and each standard field, and take the standard field with the highest first similarity as the standard field to match the source field.

[0046] In this embodiment, the first similarity can be semantic similarity or literal similarity. By calculating the first similarity between the source field and each standard field, and selecting the standard field with the highest first similarity as the standard field that matches the source field, it can be ensured that a single source field corresponds to a unique standard field, thereby improving the overall accuracy of field matching.

[0047] It should be noted that the highest first similarity score must be greater than or equal to a pre-set similarity threshold, such as 0.7, to ensure matching accuracy. If the highest first similarity score is less than the pre-set similarity threshold, manual verification is required.

[0048] In one embodiment of this application, calculating the first similarity between the source field and each standard field includes: Calculate the literal similarity and semantic similarity between the source field and each standard field; The literal similarity and semantic similarity corresponding to each standard field are weighted and summed to obtain the first similarity between the source field and each standard field.

[0049] In this embodiment, literal similarity can focus on the surface character features of fields, quantifying the degree of matching between fields through character-level algorithms. Specifically, existing Levenstein edit distance or Jaccard coefficients can be used to calculate the literal similarity between each source field and each standard field. Individual literal similarity is easily affected by field naming format. For example, "device code" and "device_id" have significant literal differences and are easily judged as having low similarity.

[0050] Semantic similarity calculation can be achieved using Natural Language Processing (NLP) techniques. First, fields are converted into low-dimensional semantic vectors, and then the semantic similarity between the corresponding semantic vectors is calculated using methods such as cosine similarity and Euclidean distance. However, individual semantic similarity calculations are easily affected by ambiguous fields.

[0051] Therefore, this embodiment performs a weighted sum of the literal similarity and semantic similarity corresponding to each standard field to obtain the first similarity between the source field and each standard field. This can comprehensively consider both literal matching and semantic matching dimensions, thereby improving the accuracy and comprehensiveness of field matching.

[0052] In one embodiment of this application, for each standard field, the literal similarity between the source field and the standard field is calculated, including: Use one of the source field and the standard field as the first string, and the other field as the second string; Construct an initial array; the j-th element of the initial array is used to represent the shortest edit distance for converting an empty string into the first j characters of the second string; 0 ≤ j ≤ n, where n is the length of the second string; Perform m iterations based on the initialized first array; where m is the length of the first string. In the i-th iteration, based on each element in the current first array, determine the shortest edit distance for converting the first i characters in the first string into the first j characters in the second string, and obtain the second array; In the first iteration, the initialized first array is used as the current first array. In subsequent iterations, the second array obtained from the previous iteration is used as the current first array. i is a natural number, and i ≤ m. The last element of the second array obtained from the nth iteration is used as the literal similarity between the source field and the standard field.

[0053] In this embodiment, the Levenshtein Distance algorithm can be used to calculate the literal similarity between the source field and the standard field. Assuming the source field is "main transformer" and the standard field is "main transformer", the literal similarity calculation process is as follows: The source field is set as the first string A: "main variable", A[0] = "main", A[1] = "variable" (length m = 2); The standard field is used as the second string B: "main transformer", B[0] = "main", B[1] = "transformer", B[2] = "pressure", B[3] = "transformer" (length n = 4); Existing Levenstein distance algorithms typically require constructing an m×n dynamic programming (DP) data table, with a computational complexity of O(m×n). In this embodiment, to optimize memory usage, a two-row dynamic programming array method is used, retaining only the current row and the previous row, thereby reducing the space complexity from O(m×n) to O(min(m, n)), which is suitable for processing longer strings.

[0054] Step 1: Initialize the first array: The first array `prev[]` (used to represent the previous line) is initialized to a length of n+1. The j-th element of the initialized first array `prev[]` is used to represent the shortest edit distance for converting an empty string into the first j (0≤j≤n) characters of the second string. The elements of the initialized first array `prev[]` are shown in Table 10 below: Table 10 - Initialized First Array prev[]

[0055] Step 2: Calculate the second array curr[0] row by row, and fill the matrix with the second array curr[0]. The first iteration (i=1, processing A[0]="main"): Calculate the shortest edit distance between the first character of the first string A and the first j characters of the second string B, where curr[0] represents the shortest edit distance between the first character of the first string A and the 0th character of the second string. The first character of the first string A is A[0]="main", and the 0th character of the second string is an empty string. Therefore, curr[0] = i = 1. Based on this, the calculation formula in Table 11 below is used to update each element of the second array curr[ ] in turn: Table 11 - Iteration Process of Round 1

[0056] In Table 11 above, Prev[j]+1 represents a deletion operation, which involves moving from the first i characters of A to the first j characters of B, and then deleting the i-th character of A. Cur[j-1]+1 represents an insertion operation, which involves moving from the first i characters of A to the first j-1 characters of B, and then inserting the j-th character of B. Prev[jl]+cost represents a substitution operation. If the i-th character of A and the j-th character of B are the same, then cost=0 (i.e., a match, no operation is needed); otherwise, cost=1 (a character is replaced).

[0057] Based on the results of the first round of iterations, each element of the second array curr[] can be obtained, as shown in Table 12 below: Table 12 - The second array curr[] obtained in the first iteration

[0058] Update: prev = curr, preparing for the next iteration.

[0059] The second iteration (i=2, processing A[1]="change"): curr[0]= i = 2, the iteration process is shown in Table 13 below: Table 13 - Second Round Iteration Process

[0060] Based on the results of the second round of iterations, each element of the second array curr[] can be obtained, as shown in Table 14 below: Table 14 - The second array curr[] obtained in the second round of iteration

[0061] The iteration operation has now ended. Result interpretation: The value in the bottom right corner of the array is 2.

[0062] Therefore, the Levenstein distance between "main transformer" and "main transformer" is 2. This intuitively corresponds to inserting the operations "voltage" and "transformer" after "main transformer".

[0063] Based on this, the literal similarity between the two fields "main transformer" and "main transformer transformer" can be calculated:

[0064] in, =2, thus we get:

[0065] That is, the literal similarity between the two fields "main transformer" and "main transformer" is 0.5.

[0066] In one embodiment of this application, the capabilities of large language models such as GPT and DeepSeek in semantic understanding and natural language parsing can be utilized to apply prompt word technology to achieve semantic similarity calculation.

[0067] The semantic similarity prompts are as follows:

[0068] The following is an example of the semantic similarity calculation results between "main transformer" and "main transformer":

[0069] In one embodiment of this application, the fusion of multi-source heterogeneous data based on the transformed source data includes: For every two transformed source data, calculate the second similarity between the two transformed source data; If the second similarity between the two transformed source data is greater than or equal to the similarity threshold, the two transformed source data are fused. If the second similarity between the two transformed source data is less than the similarity threshold, an error message is output to indicate that manual verification is required.

[0070] In this embodiment, for every two transformed source data, before data fusion, a second similarity between the two transformed source data can be calculated again. If the second similarity is greater than a preset similarity threshold, data fusion is performed; otherwise, if the second similarity is less than or equal to the preset similarity threshold, it indicates that the consistency between the two transformed source data is still poor, and an error message can be output to indicate manual verification.

[0071] In one embodiment of this application, for every two transformed source data, a second similarity is calculated between the two transformed source data, including: Calculate the structural similarity between the two transformed source data, and use this structural similarity as the second similarity between the two transformed source data; The calculation of the structural similarity between the two transformed source data includes: Data extraction is performed on the two transformed source data to obtain the first transformed metadata and the second transformed metadata. The first evaluation value is determined based on the maximum number of fields contained in the first transformation metadata and the second metadata, and the minimum number of fields contained in the first transformation metadata and the second metadata. The second evaluation value is determined based on the intersection of fields in the first transformed metadata and the second metadata, and the union of fields in the first transformed metadata and the second metadata; The third evaluation value is determined based on the nesting depth of the first and second metadata. The first, second, and third evaluation values ​​are weighted and summed to obtain the structural similarity between the two transformed source data.

[0072] In this embodiment, for every two transformed source data, the structural similarity between the two transformed source data can be calculated, and the structural similarity is used as the second similarity between the two transformed source data.

[0073] Specifically, data can be extracted from the two transformed source data to obtain the first transformed metadata and the second transformed metadata. Based on this, the similarity between the two data structures can be evaluated from multiple dimensions such as the number of fields, field types, and nesting depth.

[0074] For example, the structural similarity between two transformed source data can be calculated using the following formula: ; in, and These are the number of fields in source data y1 and source data y2, respectively. It refers to the number of common fields. The number of the union of all fields. Indicates the nesting depth of the source data y1. Indicates the nesting depth of the source data y2. , , It is a weight, and + + =1.

[0075] For example, there are two transformed source data as follows:

[0076] Step 1: Extract metadata information

[0077] Step 2: Calculate the various indicators: Number of fields:

[0078] Common fields: device_id, device_name, location; = 3; Field union: device_id, device_name, voltage, power, status, location; = 6; Nesting depth:

[0079] Step 3: Substitute into the formula to calculate: Assume weight allocation: =0.4, =0.5, =0.1,

[0080] Interpretation of Results: The two data structures have a similarity of 0.67, while the preset similarity threshold is 0.7, indicating a low degree of structural similarity. They share core fields such as device ID, device name, and location information, but differ in specific technical parameters (voltage, power) and status information, requiring manual judgment to determine whether data fusion is necessary.

[0081] In one embodiment of this application, for every two transformed source data, a second similarity is calculated between the two transformed source data, including: Calculate the instance similarity between the two transformed source data, and use this instance similarity as the second similarity between the two transformed source data; The calculation of instance similarity between the two transformed source data includes: Feature extraction is performed on the two transformed source data to obtain the first feature vector and the second feature vector; Calculate the Euclidean distance between the first and second feature vectors to obtain the second similarity between the two transformed source data.

[0082] In this embodiment, for every two transformed source data, the instance similarity between the two transformed source data can also be calculated, and the instance similarity is used as the second similarity between the two transformed source data.

[0083] Specifically, standardization or normalization methods can be used to preprocess the two transformed source data to eliminate the influence of differences in feature scales and distributions, transforming the values ​​of different features to a unified and comparable scale. The most common standardization method is Z-Score standardization.

[0084] Where: x is the original data value of a feature in a certain instance, μ is the mean of the feature (e.g., device name) in the instance, and σ is the standard deviation of the feature in the instance.

[0085] For example, we have a feature called "device runtime". Device A has a runtime of 100 hours, device B has a runtime of 200 hours, and the average runtime of all devices is 150 hours with a standard deviation of 50 hours.

[0086] Then the Z-Score of device A is:

[0087] The Z-Score of device B is:

[0088] After Z-score standardization, we use Euclidean distance again for calculation: After converting all features to Z-Score, the similarity between two instances AA and BB can be calculated using the following formula:

[0089] in, This represents the Z-Score of instance A on the k-th feature; This represents the Z-Score of instance B on the k-th feature, where k represents the number of features.

[0090] Furthermore, when new heterogeneous data sources need to be integrated, only new schema mappings and data transformation rules need to be built for them, without modifying existing fusion processes and standardized data models, thus enabling rapid system expansion. When the schema of a data source changes, only the corresponding data bridge configuration needs to be updated, ensuring system maintainability.

[0091] In summary, compared with existing technologies, the multi-source heterogeneous data fusion method based on bridging technology proposed in this embodiment has the following significant advantages: (1) Reduce integration complexity and cost: By introducing standardized data models and data bridges, the complex problem of connecting many-to-many data sources is simplified into multiple one-to-one source-to-bridge mapping problems, which greatly reduces the complexity of system integration and development and maintenance costs.

[0092] (2) Effectively solves the problem of semantic heterogeneity: The semantic mapping engine design in the bridging conversion can effectively identify and process the semantic differences between different data sources, ensuring the consistency of data at the business meaning level and improving the accuracy and availability of fused data.

[0093] (3) Improve data quality: The built-in data conversion and loading verification mechanism can clean, standardize and verify the data during the fusion process, effectively ensuring the quality of the final fused data.

[0094] (4) Enhance the scalability and flexibility of the system: This method provides a loosely coupled integration framework. When a new data source is added or the structure of an existing data source changes, only the corresponding "bridge" configurations such as database connection, single data access volume, and timeout time need to be adjusted, which has little impact on the overall system and can adapt to rapidly changing business needs.

[0095] (5) Improve data fusion efficiency: Through semi-automated pattern mapping recommendation and configurable transformation rules, manual coding work is reduced and the implementation cycle of data fusion projects is shortened.

[0096] Please refer to Figure 3 Based on the above methods, this embodiment proposes a multi-source heterogeneous data fusion system based on bridging technology. Adopting the core concept of "bridging," it designs and constructs a multi-layered architecture including a data source layer, an access adaptation layer, a bridging conversion layer, a data fusion layer, and an application service layer. This enables heterogeneous data to be smoothly connected to a unified, standardized data model, thereby achieving seamless data fusion. The technical solutions for each layer are described in detail below: Data source layer: This layer contains various heterogeneous data sources that need to be integrated, such as internal relational databases (MySQL, Oracle), business log files, sensor data streams (MQTT) on IoT platforms, public Web APIs, and unstructured document (PDF, Word) and image data repositories.

[0097] Access adaptation layer: It adopts the adapter pattern to provide dedicated pluggable access components for structured data, semi-structured data and unstructured data types, and realizes high-concurrency access through data source connection pool management; it provides a unified data acquisition interface to the outside world and shields the differences of the underlying data source.

[0098] The bridging and transformation layer (core layer) provides modules such as metadata management, data bridge, and semantic mapping engine to realize functions such as extracting metadata information from heterogeneous data sources, bridging and transforming different data formats, semantic alignment of heterogeneous data, and data cleaning and transformation.

[0099] Data Fusion Layer: As the execution layer, it provides entity recognition and linking, conflict detection and resolution, fusion strategy management, data quality control and fusion result storage functions, realizing the fusion access and unified storage of different data sources and data types.

[0100] Application Service Layer: Connects to upper-layer data applications, providing standardized data access interfaces (RESTful API services), data subscription and visual management interfaces, supporting the construction of multi-source heterogeneous data application scenarios.

[0101] Based on the same inventive concept, this application also provides a heterogeneous data fusion device based on bridging technology for implementing the heterogeneous data fusion method based on bridging technology described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the heterogeneous data fusion device based on bridging technology provided below can be found in the limitations of the heterogeneous data fusion method based on bridging technology described above, and will not be repeated here.

[0102] This application provides a heterogeneous data fusion device based on bridging technology, such as... Figure 4 As shown, the heterogeneous data fusion device 20 based on bridging technology includes: a data acquisition module 21, a field matching module 22, a data mapping module 23, and a data fusion module 24.

[0103] Among them, the data acquisition module 21 is used to acquire source data, extract data from the source data, and obtain metadata from the source data; The field matching module 22 is used to match each source field in the metadata with the standard fields in the preset standardized data model to obtain the standard fields that match the source field. Data mapping module 23 is used to determine mapping rules based on the matching relationship between source fields and standard fields, and to map source data to a standardized data model based on the mapping rules to obtain transformed source data; The data fusion module 24 is used to fuse multi-source heterogeneous data based on the transformed source data.

[0104] In one embodiment of this application, the field matching module 22 is specifically used for: For each source field in the metadata, match that source field with standard fields in the predefined canonical data model, including: Calculate the first similarity between the source field and each standard field, and take the standard field with the highest first similarity as the standard field to match the source field.

[0105] In one embodiment of this application, the field matching module 22 is further configured to: Calculate the literal similarity and semantic similarity between the source field and each standard field; The literal similarity and semantic similarity corresponding to each standard field are weighted and summed to obtain the first similarity between the source field and each standard field.

[0106] In one embodiment of this application, for each standard field, the field matching module 22 is further configured to: Use one of the source field and the standard field as the first string, and the other field as the second string; Construct an initial array; the j-th element of the initial array is used to represent the shortest edit distance for converting an empty string into the first j characters of the second string; 0 ≤ j ≤ n, where n is the length of the second string; Perform m iterations based on the initialized first array; where m is the length of the first string. In the i-th iteration, based on each element in the current first array, determine the shortest edit distance for converting the first i characters in the first string into the first j characters in the second string, and obtain the second array; In the first iteration, the initialized first array is used as the current first array. In subsequent iterations, the second array obtained from the previous iteration is used as the current first array. i is a natural number, and i ≤ m. The last element of the second array obtained from the nth iteration is used as the literal similarity between the source field and the standard field.

[0107] In one embodiment of this application, the data fusion module 24 is specifically used for: For every two transformed source data, calculate the second similarity between the two transformed source data; If the second similarity between the two transformed source data is greater than or equal to the similarity threshold, the two transformed source data are fused. If the second similarity between the two transformed source data is less than the similarity threshold, an error message is output to indicate that manual verification is required.

[0108] In one embodiment of this application, for every two transformed source data, the data fusion module 24 is further configured to: Calculate the structural similarity between the two transformed source data, and use this structural similarity as the second similarity between the two transformed source data; The calculation of the structural similarity between the two transformed source data includes: Data extraction is performed on the two transformed source data to obtain the first transformed metadata and the second transformed metadata. The first evaluation value is determined based on the maximum number of fields contained in the first transformation metadata and the second metadata, and the minimum number of fields contained in the first transformation metadata and the second metadata. The second evaluation value is determined based on the intersection of fields in the first transformed metadata and the second metadata, and the union of fields in the first transformed metadata and the second metadata; The third evaluation value is determined based on the nesting depth of the first and second metadata. The first, second, and third evaluation values ​​are weighted and summed to obtain the structural similarity between the two transformed source data.

[0109] In one embodiment of this application, for every two transformed source data, the data fusion module 24 is further configured to: Calculate the instance similarity between the two transformed source data, and use this instance similarity as the second similarity between the two transformed source data; The calculation of instance similarity between the two transformed source data includes: Feature extraction is performed on the two transformed source data to obtain the first feature vector and the second feature vector; Calculate the Euclidean distance between the first and second feature vectors to obtain the second similarity between the two transformed source data.

[0110] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 5The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of the data acquisition module 21, field matching module 22, data mapping module 23, and data fusion module 24 are shown.

[0111] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0112] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0113] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store preset information such as similarity thresholds and standardized data models.

[0114] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the heterogeneous data fusion method based on bridging technology provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0115] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0116] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., provided on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0117] Those skilled in the art will recognize that the modules / units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules / units, or it may be an electrical, mechanical, or other form of connection.

[0120] The modules / units described as separate components may or may not be physically separate. Similarly, the components shown as modules / units may or may not be physical modules / units; they may be located in one place or distributed across multiple network modules / units. Some or all of the modules / units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0121] Furthermore, the functional modules / units in the various embodiments of this application can be integrated into one processing module / unit, or each module / unit can exist physically separately, or two or more modules / units can be integrated into one module / unit. The integrated modules / units described above can be implemented in hardware or in the form of software functional modules / units.

[0122] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A heterogeneous data fusion method based on bridging technology, characterized in that, include: Obtain source data, extract data from the source data, and obtain metadata from the source data; For each source field in the metadata, the source field is matched with a standard field in a preset standardized data model to obtain a standard field that matches the source field. Based on the matching relationship between the source field and the standard field, a mapping rule is determined, and the source data is mapped to the standardized data model based on the mapping rule to obtain the transformed source data. For every two transformed source data, calculate the second similarity between the two transformed source data; If the second similarity between the two transformed source data is greater than or equal to the similarity threshold, the two transformed source data are fused. If the second similarity between the two transformed source data is less than the similarity threshold, an error message is output to indicate that manual verification is required. For every two transformed source data sets, calculate the second similarity between the two transformed source data sets, including: Calculate the structural similarity between the two transformed source data, and use this structural similarity as the second similarity between the two transformed source data; The calculation of the structural similarity between the two transformed source data includes: Data extraction is performed on the two transformed source data to obtain the first transformed metadata and the second transformed metadata. The first evaluation value is determined based on the maximum number of fields contained in the first transformation metadata and the second metadata, and the minimum number of fields contained in the first transformation metadata and the second metadata. The second evaluation value is determined based on the intersection of the fields in the first transformed metadata and the second metadata, and the union of the fields in the first transformed metadata and the second metadata. A third evaluation value is determined based on the nesting depth of the first and second metadata; The first evaluation value, the second evaluation value, and the third evaluation value are weighted and summed to obtain the structural similarity between the two transformed source data.

2. The heterogeneous data fusion method based on bridging technology as described in claim 1, characterized in that, For each source field in the metadata, the source field is matched with standard fields in a preset canonical data model, including: Calculate the first similarity between the source field and each standard field, and take the standard field with the highest first similarity as the standard field to match the source field.

3. The heterogeneous data fusion method based on bridging technology as described in claim 2, characterized in that, The calculation of the first similarity between the source field and each standard field includes: Calculate the literal similarity and semantic similarity between the source field and each standard field; The literal similarity and semantic similarity corresponding to each standard field are weighted and summed to obtain the first similarity between the source field and each standard field.

4. The heterogeneous data fusion method based on bridging technology as described in claim 3, characterized in that, For each standard field, calculate the literal similarity between the source field and the standard field, including: Use one of the source field and the standard field as the first string, and the other field as the second string; Construct an initial array; the j-th element of the initial array is used to represent the shortest edit distance for converting an empty string into the first j characters of the second string; 0 ≤ j ≤ n, where n is the length of the second string; Perform m iterations based on the initialized first array; where m is the character length of the first string; In the i-th iteration, based on each element in the current first array, determine the shortest edit distance for converting the first i characters in the first string into the first j characters in the second string, and obtain the second array; In the first iteration, the initialized first array is used as the current first array; in subsequent iterations, the second array obtained from the previous iteration is used as the current first array; i is a natural number, and i≤m; The last element of the second array obtained from the nth iteration is used as the literal similarity between the source field and the standard field.

5. A heterogeneous data fusion device based on bridging technology, characterized in that, include: The data acquisition module is used to acquire source data, extract data from the source data, and obtain metadata from the source data. The field matching module is used to match each source field in the metadata with a standard field in a preset standardized data model to obtain a standard field that matches the source field. The data mapping module is used to determine mapping rules based on the matching relationship between the source field and the standard field, and to map the source data to the standardized data model based on the mapping rules to obtain the transformed source data. The data fusion module is used for: For every two transformed source data, calculate the second similarity between the two transformed source data; If the second similarity between the two transformed source data is greater than or equal to the similarity threshold, the two transformed source data are fused. If the second similarity between the two transformed source data is less than the similarity threshold, an error message is output to indicate that manual verification is required. For every two transformed source data sets, the data fusion module is specifically used for: Calculate the structural similarity between the two transformed source data, and use this structural similarity as the second similarity between the two transformed source data; The calculation of the structural similarity between the two transformed source data includes: Data extraction is performed on the two transformed source data to obtain the first transformed metadata and the second transformed metadata. The first evaluation value is determined based on the maximum number of fields contained in the first transformation metadata and the second metadata, and the minimum number of fields contained in the first transformation metadata and the second metadata. The second evaluation value is determined based on the intersection of fields in the first transformed metadata and the second metadata, and the union of fields in the first transformed metadata and the second metadata; The third evaluation value is determined based on the nesting depth of the first and second metadata. The first, second, and third evaluation values ​​are weighted and summed to obtain the structural similarity between the two transformed source data.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 4.

Images