A multi-source heterogeneous data adaptive fusion and management method

Abstract

The application provides a multi-source heterogeneous data adaptive fusion and management method, relates to the field of data processing, and comprises the following steps: sampling data sources and analyzing sample data; extracting structural features and semantic features of the sample data to obtain data source fingerprints, and determining reference data sources; identifying fields and inferring types of the sample data to obtain metadata; performing semantic matching based on the metadata and preset target fields to generate a first mapping scheme, and performing feature matching calculation to generate a second mapping scheme; fusing the first mapping scheme and the second mapping scheme to obtain field mapping rules and comprehensive divergence; performing data conversion on the data sources according to the field mapping rules and performing quality evaluation to obtain quality scores; and adjusting a sensitivity coefficient according to the quality scores and the comprehensive divergence. The application realizes strategy migration and parameter adaptive optimization based on historical experience, and improves the automation degree and conversion quality of heterogeneous data fusion.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for adaptive fusion and governance of multi-source heterogeneous data. Background Technology

[0002] With the deepening of digital transformation, various information systems have generated massive amounts of multi-source heterogeneous data. This data is distributed across different system platforms, exhibiting diverse formats, structural differences, and inconsistent naming conventions, posing significant challenges to data integration. The access and transformation of heterogeneous data sources typically rely on manually configuring mapping rules. Technical personnel must analyze the structural characteristics and business semantics of each data source individually, manually writing field mapping relationships and transformation scripts. This approach is not only inefficient but also ill-suited to the rapid access requirements of new data sources.

[0003] Existing data integration methods primarily rely on predefined mapping templates or fixed matching strategies. When the structure of a new data source differs from existing templates, manual reconfiguration is required, lacking adaptability. Furthermore, during field mapping, a single matching strategy struggles to account for the characteristic differences between different data sources, easily leading to mapping errors or insufficient confidence. For parameter configuration during rule generation, existing methods typically use fixed parameters or start optimization from default values, failing to leverage historical processing experience, resulting in slow parameter convergence and a low success rate for initial processing of new data sources.

[0004] Chinese invention patent CN111190881A discloses a data governance method and system. This method establishes a data standard library by cleaning and fusing data sources, uses a meta-model to collect metadata from the data standard library for governance, performs standardized data transformation, and finally implements data quality management and regulatory protection. This patent primarily focuses on the standardization and quality management of the data cleaning and fusion process. It achieves standardized data transformation by establishing a data standard system and adopts a layered processing approach of pre-cleaning and post-cleaning to grade and evaluate data quality. However, when dealing with multi-source data access, it lacks a dynamic adjustment mechanism during processing and struggles to automatically optimize mapping strategies based on the characteristics and differences of the data sources. Summary of the Invention

[0005] In view of this, the present invention provides an adaptive fusion and governance method for multi-source heterogeneous data, which solves the technical problem of the lack of adaptive adjustment of mapping strategies in the prior art, realizes strategy migration and parameter adaptive optimization based on historical experience, and improves the automation level and transformation quality of heterogeneous data fusion.

[0006] The technical solution of this invention is implemented as follows: This invention provides a method for adaptive fusion and governance of multi-source heterogeneous data, comprising: S1. Obtain the data source, sample and parse the data source to obtain sample data; S2. Extract structural and semantic features from the sample data to obtain the data source fingerprint. Compare the data source fingerprint with the historical data source fingerprints in the fingerprint database, calculate the structural similarity and semantic similarity respectively, and determine the reference data source. S3. Based on structural similarity and semantic similarity, perform field identification and type inference on the sample data to obtain metadata, and initialize the sensitivity coefficient based on structural similarity and semantic similarity. S4. Based on metadata and preset target fields, perform semantic matching to generate a first mapping scheme, and perform feature matching calculation to generate a second mapping scheme. Determine the fusion weight based on structural similarity and semantic similarity, and fuse the first mapping scheme and the second mapping scheme based on the fusion weight to obtain the field mapping rule and comprehensive divergence degree. S5. Based on the field mapping rules, perform data transformation on the data source and conduct quality assessment to obtain a quality score. Adjust the sensitivity coefficient based on the quality score and the overall divergence. When the quality score and the overall divergence reach the preset requirements, store the data source fingerprint and field mapping rules in the fingerprint database.

[0007] Based on the above technical solutions, preferably, step S2 specifically includes: Structural features are extracted from the sample data, including statistical information on data format type, number of fields, field length, and character type distribution characteristics, to form a structural feature vector; Semantic feature extraction is performed on the sample data to obtain word frequency vectors, semantic embedding vectors, and business feature vectors, forming a semantic feature vector; The structural feature vector and semantic feature vector are used as the fingerprint of the current data source. The structural similarity and semantic similarity between the fingerprint of the current data source and the fingerprint of the historical data source stored in the fingerprint database are calculated respectively. The comprehensive similarity is calculated based on the structural similarity and semantic similarity, and the historical data source with the highest comprehensive similarity is selected as the reference data source. The comprehensive similarity is the geometric mean of the structural similarity and semantic similarity.

[0008] Based on the above technical solutions, preferably, step S3 specifically includes: The metadata extraction strategy is determined based on structural similarity, semantic similarity, a first similarity threshold, and a second similarity threshold; wherein the first similarity threshold is greater than the second similarity threshold. For each identified field, data type and field constraint relationship are identified to obtain a field list, field data type, field constraint conditions, and field business meaning, thus forming metadata; The sensitivity coefficient is initialized based on structural and semantic similarity: When both structural similarity and semantic similarity are higher than the first similarity threshold, the optimal sensitivity coefficient of the reference data source is read from the fingerprint database as the initial sensitivity coefficient. When at least one of the structural similarity or semantic similarity is below the first similarity threshold but above the second similarity threshold, the multi-case strategy fusion mechanism is activated to calculate the initial sensitivity coefficient. When at least one of the structural similarity or semantic similarity is lower than the second similarity threshold, the preset default sensitivity coefficient is used as the initial sensitivity coefficient.

[0009] Based on the above technical solutions, preferably, the metadata extraction strategy specifically includes: When both structural similarity and semantic similarity are higher than the first similarity threshold, the metadata of the reference data source will be compared with the field list of the current data source to identify newly added fields, missing fields, and matching fields. When at least one of the structural similarity or semantic similarity is below the first similarity threshold but above the second similarity threshold, incremental analysis and identification of the current data source is performed based on the metadata of the reference data source. When at least one of the structural similarity or semantic similarity is below the second similarity threshold, format recognition and type inference are performed on all fields of the current data source.

[0010] Based on the above technical solutions, preferably, the multi-case strategy fusion mechanism includes: Select the top K historical data sources from the fingerprint database that rank highest in overall similarity to the current data source. Extract the optimal sensitivity coefficient and path divergence of each historical data source. Select the historical data source with the lowest path divergence as the baseline case. Based on the optimal sensitivity coefficient of the baseline case, adjust the current data source according to the similarity difference between the baseline case and the baseline case to obtain the initial sensitivity coefficient. The calculation formula is as follows: ; in, This represents the initial sensitivity coefficient of the current data source; The optimal sensitivity coefficient represents the baseline case. To explore coefficients; Indicates differences in similarity; This represents the standard deviation of the sensitivity coefficients for the first K historical cases.

[0011] Based on the above technical solutions, preferably, the semantic matching scheme for generating the first mapping based on metadata and preset target fields specifically includes: Obtain a preset target data model, which contains several target fields and each target field has a standardized field name and business meaning description; For each source data field in the metadata, calculate the similarity of its field name with each target field in the target data model; For each source data field, the target field with the highest similarity to its field name is selected as the semantic matching recommendation mapping target, and the similarity of the field name is used as the first confidence level to obtain the first mapping scheme and its corresponding first confidence level.

[0012] Based on the above technical solutions, preferably, the method for generating the second mapping by matching data features specifically includes: For each source data field in the metadata, extract data type features, numerical distribution features, and pattern matching features from the metadata as data feature vectors; Calculate the feature similarity between the feature vector of each source data field and the feature vector of each target field in the target data model; For each source data field, the target field with the highest feature similarity is selected as the mapping target recommended for data feature matching. The feature similarity is used as the second confidence level to obtain the second mapping scheme and its corresponding second confidence level.

[0013] Based on the above technical solutions, preferably, the formula for calculating the comprehensive divergence degree is: ; ; ; in, Indicates the overall degree of divergence; This indicates the total number of fields in the source data; Indicates a field index; Indicates the first The divergence flag for each field is set to 1 if the target fields recommended by the two paths are inconsistent, and 0 otherwise. Indicates the first The absolute difference between the first and second confidence levels for each field. The weights represent semantic matching. The weights representing the matching of data features; This is the sensitivity coefficient; Indicates structural similarity; Indicates semantic similarity.

[0014] Based on the above technical solutions, preferably, step S5 specifically includes: The data source is transformed according to the field mapping rules to obtain the transformed data; the data transformation includes data type conversion, format standardization and unit conversion operations. The transformed data is subjected to a quality assessment to obtain a quality score. The quality assessment includes completeness assessment, accuracy assessment and consistency assessment. The completeness assessment checks whether the transformed data records contain all the required fields required by the target model and counts the proportion of missing fields. The accuracy assessment verifies whether the transformed field values ​​conform to the data type, value range and business logic constraints of the target fields through sampling. The consistency assessment checks whether the transformed data meets the constraint relationships between fields. The sensitivity coefficient is adjusted based on the quality score and overall divergence. When the quality score is lower than the first quality threshold and the overall divergence is higher than the first divergence threshold, the negative adjustment of the sensitivity coefficient is calculated and the sensitivity coefficient is updated, where the second quality threshold is greater than the first quality threshold and the first divergence threshold is greater than the second divergence threshold. When the quality score is higher than the second quality threshold and the overall divergence is lower than the second divergence threshold, the positive adjustment of the sensitivity coefficient is calculated and the sensitivity coefficient is updated. When the quality score and overall divergence reach the preset requirements, the data source fingerprint, metadata, optimal sensitivity coefficient, and field mapping rules of the current data source are stored in the fingerprint database.

[0015] More preferably, the formula for adjusting the sensitivity coefficient is as follows: When the quality score is below the first quality threshold and the overall divergence is above the first divergence threshold, the formula for calculating the negative adjustment is: ; When the quality score is higher than the second quality threshold and the overall divergence is lower than the second divergence threshold, the formula for calculating the positive adjustment is: ; in, This indicates the adjustment amount for the sensitivity coefficient; This is the learning rate coefficient; Indicates the overall degree of divergence; Score the target quality; Indicates the current quality score; This is the second quality threshold.

[0016] The present invention has the following advantages over the prior art: (1) By extracting structural and semantic features from the sample data in two dimensions, a unique identifier for the data source is formed, and the similarity is calculated by comparing it with the fingerprints of historical data sources in the fingerprint database. This automatically determines the reference data source, realizes strategy migration and parameter adaptive optimization, and improves the automation and conversion quality of heterogeneous data fusion. (2) By dynamically adjusting the mapping strategy weights based on the differences in structural similarity and semantic similarity, it is possible to more accurately adapt to data sources with different features, thereby improving the accuracy of field mapping and the quality of conversion. (3) Ensure the reliability of migration parameters through a multi-case strategy fusion mechanism, avoid learning from erroneous experiences from unstable cases, and improve the accuracy of parameter initialization. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a method for adaptive fusion and governance of multi-source heterogeneous data according to the present invention; Figure 2 This is a flowchart of data source fingerprint extraction and matching for a multi-source heterogeneous data adaptive fusion and governance method according to the present invention. Figure 3 This is a schematic diagram of the field mapping dual-path fusion architecture of the adaptive fusion and governance method for multi-source heterogeneous data of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] like Figure 1 As shown, this invention provides a method for adaptive fusion and governance of multi-source heterogeneous data, including: S1. Obtain the data source, sample and parse the data source to obtain sample data; Understandably, when a new data source is connected, a connection channel is first established with that data source, and the appropriate access protocol is selected based on the type of data source. For database-type data sources, access is made through standard database connection protocols such as JDBC or ODBC; for file-type data sources, access is made through file transfer protocols such as FTP, SFTP, or HTTP; and for API interface-type data sources, access is made through RESTful API or SOAP interfaces.

[0021] In one embodiment of the present invention, the sample data is extracted using an adaptive sampling strategy, specifically: Obtain metadata about the data source, including basic information such as the total amount of data, the number of data tables or files, and the data update frequency; Based on the sampling ratio of metadata, the calculation formula is as follows: in, Indicates the sampling ratio; This indicates the total number of records in the data source, expressed in rows. After determining the sampling ratio, a stratified random sampling method is used to draw samples: For structured data, uniform sampling is performed based on the primary key or timestamp field of the data table to ensure that the sample covers the time span of the data and the business cycle; For semi-structured or unstructured data, interval sampling is performed at different locations in the data file to avoid sampling bias caused by local features of the data; After sampling, the extracted sample data is initially analyzed to identify its basic data format type, including but not limited to common formats such as CSV, JSON, XML, fixed-length text, and delimited text. The data encoding method, such as character set encoding such as UTF-8 and GBK, is also detected. The analysis results are stored as a temporary snapshot of the original data as input for subsequent fingerprint extraction and format recognition.

[0022] In one embodiment of the present invention, the sampling ratio The range of values ​​is When the data source is small, it can collect a sufficient number of samples. When the data source has less than one million records, the sampling ratio is automatically increased to ensure that at least 10,000 sample data are collected.

[0023] S2. Extract structural and semantic features from the sample data to obtain the data source fingerprint. Compare the data source fingerprint with historical data source fingerprints in the fingerprint database, calculate the structural similarity and semantic similarity respectively, and determine the reference data source.

[0024] This invention extracts structural and semantic features from sample data to form a unique identifier for the data source. It then compares the identifier with the fingerprints of historical data sources in the fingerprint database to calculate the similarity, thereby automatically determining the reference data source. This invention can leverage historical processing experience to achieve strategy migration and adaptive parameter optimization, improving the automation level and conversion quality of heterogeneous data fusion.

[0025] like Figure 2 As shown, specifically, step S2 includes: Structural features are extracted from the sample data, including statistical information on data format type, number of fields, field length, and character type distribution characteristics, to form a structural feature vector; Semantic feature extraction is performed on the sample data to obtain word frequency vectors, semantic embedding vectors, and business feature vectors, forming a semantic feature vector; Using the structural and semantic feature vectors as the fingerprints of the current data source, the structural and semantic similarities between the current data source fingerprint and the historical data source fingerprints stored in the fingerprint database are calculated. A comprehensive similarity is then calculated based on the structural and semantic similarities, and the historical data source with the highest comprehensive similarity is selected as the reference data source. The comprehensive similarity is the geometric mean of the structural and semantic similarities. in, Indicates structural similarity. This represents the structural feature vector of the newly connected data source; This represents the structural feature vector of the reference data source in the fingerprint database; This represents the vector dot product operation; The Euclidean norm of a vector; This indicates the lexical similarity of the field names. Represents the similarity in the semantic embedding space. This represents the word weight coefficient.

[0026] Understandably, cosine similarity is used to measure the similarity between two structural feature vectors, with values ​​ranging from [value range missing]. The larger the value, the more similar the structures. The range of values ​​is It is obtained by calculating the Jaccard similarity coefficient of the field name sets of two data sources, that is, the size of the intersection of the two field name sets divided by the size of the union. The range of values ​​is It is obtained by calculating the cosine similarity of the semantic embedding vectors of the two data sources. The range of values ​​is This is used to balance the contributions of lexical matching and semantic matching.

[0027] This invention dynamically adjusts the mapping strategy weights based on differences in structural and semantic similarity, enabling more accurate adaptation to data sources with different characteristics and improving the accuracy and quality of field mapping.

[0028] In one embodiment of the present invention, structural feature extraction specifically includes: The data includes information on the format of the statistical sample data, such as calculating the hash value of the data format type as a format identifier, the total number of fields, the average length and standard deviation of each field, the proportion of numeric characters in field values, and the frequency of special characters such as commas, semicolons, and colons. Analyze the data type distribution of each field, determine the candidate data type of the field through regular expression matching and statistical inference, such as integer, floating point, date, string, etc., and record the confidence level of each type; For numeric fields, the system calculates the minimum, maximum, median, and quartiles of their value range, as well as statistical characteristics such as skewness and kurtosis of the numerical distribution. For string fields, the system statistically analyzes the distribution characteristics of field length and the diversity of character sets; All structural features are organized into a high-dimensional vector, denoted as the structural feature vector. .

[0029] In one embodiment of the present invention, semantic feature extraction specifically includes: Extract all field names from the sample data and perform standardized preprocessing on the field names, including case normalization, unified conversion of underscore and camelCase naming, and stop word filtering. The term frequency-inverse document frequency (TF-IDF) method is used to calculate the feature weights of field names and construct the term frequency vectors of field names; By using a pre-trained word vector model, field names are mapped to a semantic space to obtain the semantic embedding representation of the field names; For field names containing Chinese characters, word segmentation is performed before semantic encoding; Identify business keywords and match business terms in field names and field contents according to a predefined industry thesaurus, such as terms like account balance and transaction amount in the financial field, and terms like call duration and data consumption in the telecommunications field. Analyze the frequency and co-occurrence patterns of various business keywords to construct business semantic features; The field name vector, semantic embedding vector, and business feature vector are concatenated to form the semantic feature vector. .

[0030] S3. Based on structural similarity and semantic similarity, perform field identification and type inference on the sample data to obtain metadata, and initialize the sensitivity coefficient based on structural similarity and semantic similarity. Specifically, step S3 includes: The metadata extraction strategy is determined based on structural similarity, semantic similarity, a first similarity threshold, and a second similarity threshold; wherein the first similarity threshold is greater than the second similarity threshold. For each identified field, data type and field constraint relationship are identified to obtain a field list, field data type, field constraint conditions, and field business meaning, thus forming metadata; The sensitivity coefficient is initialized based on structural and semantic similarity: When both structural similarity and semantic similarity are higher than the first similarity threshold, the optimal sensitivity coefficient of the reference data source is read from the fingerprint database as the initial sensitivity coefficient: in This represents the initial sensitivity coefficient of the current data source; This represents the optimal sensitivity coefficient obtained by optimizing the reference data source through quality feedback in historical processing, and this value is stored in the fingerprint database; When at least one of the structural similarity or semantic similarity is below the first similarity threshold but above the second similarity threshold, the multi-case strategy fusion mechanism is activated to calculate the initial sensitivity coefficient. When at least one of the structural similarity or semantic similarity is below the second similarity threshold, a preset default sensitivity coefficient is used as the initial sensitivity coefficient. in The default sensitivity coefficient is preset and is usually set to a value of [value missing]. .

[0031] In one embodiment of the present invention, the metadata extraction strategy specifically includes: When both structural similarity and semantic similarity are higher than the first similarity threshold, the metadata of the reference data source will be compared with the field list of the current data source to identify newly added fields, missing fields, and matching fields. When at least one of the structural similarity or semantic similarity is below the first similarity threshold but above the second similarity threshold, incremental analysis and identification of the current data source is performed based on the metadata of the reference data source. When at least one of the structural similarity or semantic similarity is below the second similarity threshold, format recognition and type inference are performed on all fields of the current data source.

[0032] Understandably, for matching fields, the reference metadata is directly reused; for newly added fields, only these incremental fields undergo detailed format identification and type inference; for missing fields, they are marked as optional fields and handled for compatibility in subsequent conversions.

[0033] In one embodiment of the present invention, data type identification specifically includes: For numeric fields, determine whether they are integers or floating-point numbers, whether they contain negative numbers, and what precision requirements they have; For date and time fields, identify their specific date format pattern, such as the order of year, month, day, hour, minute, and second, the type of separator, and whether time zone information is included. For string fields, analyze whether they have a fixed length and whether they follow specific encoding rules such as ID card number format, telephone number format, etc.

[0034] Furthermore, the multi-case strategy integration mechanism specifically includes: Select the top K historical data sources from the fingerprint database that rank highest in overall similarity to the current data source. Extract the optimal sensitivity coefficient and path divergence of each historical data source. Select the historical data source with the lowest path divergence as the baseline case. Based on the optimal sensitivity coefficient of the baseline case, adjust the current data source according to the similarity difference between the baseline case and the baseline case to obtain the initial sensitivity coefficient. The calculation formula is as follows: ; in, This represents the initial sensitivity coefficient of the current data source; The optimal sensitivity coefficient represents the baseline case. To explore coefficients; Indicates differences in similarity; This represents the standard deviation of the sensitivity coefficients for the first K historical cases.

[0035] This invention ensures the reliability of migration parameters through a multi-case strategy fusion mechanism, avoids learning erroneous experiences from cases with unstable quality, and improves the accuracy of parameter initialization.

[0036] like Figure 3 As shown, S4 generates a first mapping scheme by semantic matching based on metadata and preset target fields, and generates a second mapping scheme by feature matching calculation. The fusion weight is determined based on structural similarity and semantic similarity. The first mapping scheme and the second mapping scheme are fused based on the fusion weight to obtain the field mapping rules and comprehensive divergence degree.

[0037] In one embodiment of the present invention, a first mapping scheme is generated by semantic matching based on metadata and a preset target field, specifically including: Obtain a preset target data model, which contains several target fields and each target field has a standardized field name and business meaning description; For each source data field in the metadata, calculate the similarity of its field name with each target field in the target data model; For each source data field, the target field with the highest similarity to its field name is selected as the semantic matching recommendation mapping target, and the similarity of the field name is used as the first confidence level to obtain the first mapping scheme and its corresponding first confidence level.

[0038] Specifically, semantic similarity calculation considers both literal matching and semantic embedding matching. in, Indicates the source field name; Indicates the target field name; This represents the edit distance between two strings, which is the minimum number of single-character edit operations required to convert one string into another. Indicates the string length. A semantic embedding vector representing the source field name; A semantic embedding vector representing the target field name. This is the literal weighting coefficient, with a value range of [value range missing]. .

[0039] Understandable, for source fields The system selects the target field with the highest semantic similarity as the mapping target for semantic path recommendation, and uses this highest similarity as the confidence score of the semantic path. The range of values ​​is .

[0040] In one embodiment of the present invention, performing data feature matching to generate a second mapping scheme specifically includes: For each source data field in the metadata, data type features, numerical distribution features, and pattern matching features are extracted from the metadata as data feature vectors. Data type features describe the basic type of the field value, such as numeric, string, date, and boolean, and a type distribution vector is constructed by inferring the type from the sample data. Numerical distribution features are applicable to numeric fields and include statistics such as mean, variance, minimum, maximum, skewness, and kurtosis. Pattern matching features use regular expressions to determine whether the field value conforms to a specific business pattern, such as ID card number pattern, mobile phone number pattern, or email pattern, and calculate the matching rate for each pattern. Calculate the feature similarity between the feature vector of each source data field and the feature vector of each target field in the target data model; For each source data field, the target field with the highest feature similarity is selected as the mapping target recommended for data feature matching. The feature similarity is used as the second confidence level to obtain the second mapping scheme and its corresponding second confidence level.

[0041] In one embodiment of the present invention, for the source field and target field Calculate the similarity between their data feature vectors : in, $ represents the data feature vector of the source field; A typical data feature vector representing the target field.

[0042] Understandably, the target field with the highest similarity to the source field's data features is selected as the mapping target for the data feature path recommendation, and this highest similarity is used as the confidence level of the data feature path. The range of values ​​is .

[0043] In one embodiment of the present invention, the formula for calculating the overall divergence degree is: ; ; ; in, Indicates the overall degree of divergence; This indicates the total number of fields in the source data; Indicates a field index; Indicates the first The divergence flag for each field is set to 1 if the target fields recommended by the two paths are inconsistent, and 0 otherwise. Indicates the first The absolute difference between the first and second confidence levels for each field. The weights represent semantic matching. The weights representing the matching of data features; This is the sensitivity coefficient; Indicates structural similarity; Indicates semantic similarity.

[0044] Understandable, difference items The difference between structural similarity and semantic similarity is represented by its sign, which determines the direction of the weight bias, while its absolute value reflects the strength of the bias. When When the difference term is positive, The function outputs a positive value, making It is biased towards semantic paths; when When the difference term is negative, it makes This approach favors data feature paths. In other words, high structural similarity coupled with relatively low semantic similarity indicates a good match in the data source structure but potential naming inconsistencies. In this case, a deeper analysis of field meanings using semantic paths is needed to eliminate naming differences. Conversely, a data feature path is used to match data based on its content features. Collaborative enhancements. The overall level of similarity between two systems is represented by a higher value, indicating a more comprehensive understanding of the data source and a greater need to increase the weight polarization to make decisions more decisive. Conversely, a lower value indicates a lack of prior knowledge and a need to reduce the polarization to allow the two paths to function in a balanced manner. Parameters control the sensitivity of polarization. The larger the similarity, the more significant the impact of differences on the weights, and the more extreme the weight allocation. The smaller the value, the smoother the weight changes, maintaining a relative balance. The function was chosen because it has good saturation characteristics. When the input value is large, the output tends to be close to 1 or -1, and when the input value is close to zero, the output changes approximately linearly. This characteristic allows the weights to transition smoothly between polarization and balance, avoiding drastic changes in weights.

[0045] By introducing a comprehensive divergence degree, this invention can more accurately locate parameter mismatch problems, realize conditional triggering and targeted adjustment of parameter optimization, improve the pertinence and stability of parameter updates, and avoid misjudging path logic errors as parameter problems.

[0046] S5. Based on the field mapping rules, perform data transformation on the data source and conduct quality assessment to obtain a quality score. Adjust the sensitivity coefficient based on the quality score and the overall divergence. When the quality score and the overall divergence reach the preset requirements, store the data source fingerprint and field mapping rules in the fingerprint database.

[0047] Specifically, step S5 includes: The data source is transformed according to the field mapping rules to obtain the transformed data; the data transformation includes data type conversion, format standardization and unit conversion operations. The transformed data is subjected to a quality assessment to obtain a quality score. The quality assessment includes completeness assessment, accuracy assessment and consistency assessment. The completeness assessment checks whether the transformed data records contain all the required fields required by the target model and counts the proportion of missing fields. The accuracy assessment verifies whether the transformed field values ​​conform to the data type, value range and business logic constraints of the target fields through sampling. The consistency assessment checks whether the transformed data meets the constraint relationships between fields. The sensitivity coefficient is adjusted based on the quality score and overall divergence. When the quality score is lower than the first quality threshold and the overall divergence is higher than the first divergence threshold, the negative adjustment of the sensitivity coefficient is calculated and the sensitivity coefficient is updated, where the second quality threshold is greater than the first quality threshold and the first divergence threshold is greater than the second divergence threshold. When the quality score is higher than the second quality threshold and the overall divergence is lower than the second divergence threshold, the positive adjustment of the sensitivity coefficient is calculated and the sensitivity coefficient is updated. When the quality score and overall divergence reach the preset requirements, the data source fingerprint, metadata, optimal sensitivity coefficient, and field mapping rules of the current data source are stored in the fingerprint database.

[0048] Furthermore, the formula for adjusting the sensitivity coefficient is as follows: When the quality score is below the first quality threshold and the overall divergence is above the first divergence threshold, the formula for calculating the negative adjustment is: in This represents the updated sensitivity coefficient; This represents the sensitivity coefficient currently in use, i.e., the initial value obtained. . and The function is used to constrain the range of parameters. Within the range, avoid having parameters that are too small, resulting in no difference in weights, or parameters that are too large, resulting in excessive polarization of weights.

[0049] When the quality score is higher than the second quality threshold and the overall divergence is lower than the second divergence threshold, the formula for calculating the positive adjustment is: in, This indicates the adjustment amount for the sensitivity coefficient; This is the learning rate coefficient; Indicates the overall degree of divergence; Score the target quality; Indicates the current quality score; This is the second quality threshold.

[0050] This invention forms a closed loop of knowledge transfer across data sources through continuous accumulation and parameter updates of the fingerprint database, gradually optimizing processing efficiency and conversion quality, and ultimately achieving intelligent processing of heterogeneous data fusion.

[0051] In one embodiment of the present invention, quality scoring The calculation uses a weighted average method: in, Indicates the completeness score; Indicates the accuracy score; Indicates the consistency score; , , These are the weighting coefficients for the three dimensions.

[0052] Understandable. , , The value range can be set according to the actual usage scenario, and the value range is 0.5. And satisfy .

[0053] In one embodiment of the present invention, , , The values ​​are 0.3, 0.4 and 0.3 respectively.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for adaptive fusion and governance of multi-source heterogeneous data, characterized in that, include: S1. Obtain the data source, sample and parse the data source to obtain sample data; S2. Extract structural and semantic features from the sample data to obtain the data source fingerprint. Compare the data source fingerprint with the historical data source fingerprints in the fingerprint database, calculate the structural similarity and semantic similarity respectively, and determine the reference data source. S3. Based on structural similarity and semantic similarity, perform field identification and type inference on the sample data to obtain metadata, and initialize the sensitivity coefficient based on structural similarity and semantic similarity. S4. Based on metadata and preset target fields, perform semantic matching to generate a first mapping scheme, and perform feature matching calculation to generate a second mapping scheme. Determine the fusion weight based on structural similarity and semantic similarity, and fuse the first mapping scheme and the second mapping scheme based on the fusion weight to obtain the field mapping rule and comprehensive divergence degree. S5. Based on the field mapping rules, perform data transformation on the data source and conduct quality assessment to obtain a quality score. Adjust the sensitivity coefficient based on the quality score and the overall divergence. When the quality score and the overall divergence reach the preset requirements, store the data source fingerprint and field mapping rules in the fingerprint database.

2. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 1, characterized in that: Step S2 specifically includes: Structural features are extracted from the sample data, including statistical information on data format type, number of fields, field length, and character type distribution characteristics, to form a structural feature vector; Semantic feature extraction is performed on the sample data to obtain word frequency vectors, semantic embedding vectors, and business feature vectors, forming a semantic feature vector; The structural feature vector and semantic feature vector are used as the fingerprint of the current data source. The structural similarity and semantic similarity between the fingerprint of the current data source and the fingerprint of the historical data source stored in the fingerprint database are calculated respectively. The comprehensive similarity is calculated based on the structural similarity and semantic similarity, and the historical data source with the highest comprehensive similarity is selected as the reference data source. The comprehensive similarity is the geometric mean of the structural similarity and semantic similarity.

3. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 1, characterized in that: Step S3 specifically includes: The metadata extraction strategy is determined based on structural similarity, semantic similarity, a first similarity threshold, and a second similarity threshold; wherein the first similarity threshold is greater than the second similarity threshold. For each identified field, data type and field constraint relationship are identified to obtain a field list, field data type, field constraint conditions, and field business meaning, thus forming metadata; The sensitivity coefficient is initialized based on structural and semantic similarity: When both structural similarity and semantic similarity are higher than the first similarity threshold, the optimal sensitivity coefficient of the reference data source is read from the fingerprint database as the initial sensitivity coefficient. When at least one of the structural similarity or semantic similarity is below the first similarity threshold but above the second similarity threshold, the multi-case strategy fusion mechanism is activated to calculate the initial sensitivity coefficient. When at least one of the structural similarity or semantic similarity is lower than the second similarity threshold, the preset default sensitivity coefficient is used as the initial sensitivity coefficient.

4. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 3, characterized in that: The metadata extraction strategy specifically includes: When both structural similarity and semantic similarity are higher than the first similarity threshold, the metadata of the reference data source will be compared with the field list of the current data source to identify newly added fields, missing fields, and matching fields. When at least one of the structural similarity or semantic similarity is below the first similarity threshold but above the second similarity threshold, incremental analysis and identification of the current data source is performed based on the metadata of the reference data source. When at least one of the structural similarity or semantic similarity is below the second similarity threshold, format recognition and type inference are performed on all fields of the current data source.

5. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 4, characterized in that: The multi-case strategy fusion mechanism specifically includes: Select the top K historical data sources from the fingerprint database that rank highest in overall similarity to the current data source. Extract the optimal sensitivity coefficient and path divergence of each historical data source. Select the historical data source with the lowest path divergence as the baseline case. Based on the optimal sensitivity coefficient of the baseline case, adjust the current data source according to the similarity difference between the baseline case and the baseline case to obtain the initial sensitivity coefficient. The calculation formula is as follows: ； in, This represents the initial sensitivity coefficient of the current data source; The optimal sensitivity coefficient represents the baseline case. To explore coefficients; Indicates differences in similarity; This represents the standard deviation of the sensitivity coefficients for the first K historical cases.

6. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 1, characterized in that: The process of generating a first mapping scheme based on semantic matching of metadata and preset target fields specifically includes: Obtain a preset target data model, which contains several target fields and each target field has a standardized field name and business meaning description; For each source data field in the metadata, calculate the similarity of its field name with each target field in the target data model; For each source data field, the target field with the highest similarity to its field name is selected as the semantic matching recommendation mapping target, and the similarity of the field name is used as the first confidence level to obtain the first mapping scheme and its corresponding first confidence level.

7. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 6, characterized in that: The process of generating a second mapping scheme by matching data features specifically includes: For each source data field in the metadata, extract data type features, numerical distribution features, and pattern matching features from the metadata as data feature vectors; Calculate the feature similarity between the feature vector of each source data field and the feature vector of each target field in the target data model; For each source data field, the target field with the highest feature similarity is selected as the mapping target recommended for data feature matching. The feature similarity is used as the second confidence level to obtain the second mapping scheme and its corresponding second confidence level.

8. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 7, characterized in that: The formula for calculating the overall divergence degree is: ； ； ； in, Indicates the overall degree of divergence; This indicates the total number of fields in the source data; Indicates a field index; Indicates the first The divergence flag for each field is set to 1 if the target fields recommended by the two paths are inconsistent, and 0 otherwise. Indicates the first The absolute difference between the first and second confidence levels for each field. The weights represent semantic matching. The weights representing the matching of data features; This is the sensitivity coefficient; Indicates structural similarity; Indicates semantic similarity.

9. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 1, characterized in that: Step S5 specifically includes: The data source is transformed according to the field mapping rules to obtain the transformed data; the data transformation includes data type conversion, format standardization and unit conversion operations. The transformed data is subjected to a quality assessment to obtain a quality score. The quality assessment includes completeness assessment, accuracy assessment and consistency assessment. The completeness assessment checks whether the transformed data records contain all the required fields required by the target model and counts the proportion of missing fields. The accuracy assessment verifies whether the transformed field values ​​conform to the data type, value range and business logic constraints of the target fields through sampling. The consistency assessment checks whether the transformed data meets the constraint relationships between fields. The sensitivity coefficient is adjusted based on the quality score and overall divergence. When the quality score is lower than the first quality threshold and the overall divergence is higher than the first divergence threshold, the negative adjustment of the sensitivity coefficient is calculated and the sensitivity coefficient is updated, where the second quality threshold is greater than the first quality threshold and the first divergence threshold is greater than the second divergence threshold. When the quality score is higher than the second quality threshold and the overall divergence is lower than the second divergence threshold, the positive adjustment of the sensitivity coefficient is calculated and the sensitivity coefficient is updated. When the quality score and overall divergence reach the preset requirements, the data source fingerprint, metadata, optimal sensitivity coefficient, and field mapping rules of the current data source are stored in the fingerprint database.

10. The adaptive fusion and governance method for multi-source heterogeneous data as described in claim 9, characterized in that: The formula for adjusting the sensitivity coefficient is as follows: When the quality score is below the first quality threshold and the overall divergence is above the first divergence threshold, the formula for calculating the negative adjustment is: ； When the quality score is higher than the second quality threshold and the overall divergence is lower than the second divergence threshold, the formula for calculating the positive adjustment is: ； in, This indicates the adjustment amount for the sensitivity coefficient; This is the learning rate coefficient; Indicates the overall degree of divergence; Score the target quality; Indicates the current quality score; This is the second quality threshold.

Images