An ai multi-source data processing method and system based on big data
By constructing a field co-occurrence matrix and adjusting the adaptive similarity threshold, the problem of inaccurate identification of semantically equivalent fields in multi-source heterogeneous data environments is solved, and efficient field matching and data fusion are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGNAN INFORMATION TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-05-07
- Publication Date
- 2026-07-07
AI Technical Summary
In multi-source heterogeneous data environments, existing technologies cannot accurately identify semantically equivalent fields, resulting in a high false match rate and an inability to balance the matching recall rate of key fields with the overall recall rate.
By constructing a field co-occurrence matrix, calculating field relevance and data completeness, generating centroid vectors, calculating adaptive similarity thresholds, and combining field weights and aggregation stability, the matching thresholds are dynamically adjusted to achieve differentiated matching decisions.
While reducing the false match rate of key fields, the overall match recall rate is maintained, achieving synergistic optimization of match precision and recall rate, and improving the reliability and accuracy of field alignment.
Smart Images
Figure CN122132558B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to an AI multi-source data processing method and system based on big data. Background Technology
[0002] With the rapid advancement of digital transformation in government and enterprises, business data is generally stored across various heterogeneous data sources, including relational databases, document databases, log systems, and structured files. To meet the needs of cross-departmental data sharing, comprehensive querying, and business collaboration, it is necessary to perform schema alignment on the fields in each data source and establish accurate field mapping relationships to support subsequent data fusion and cross-source relational query operations.
[0003] To address the aforementioned field mapping requirements, a matching method based on the literal similarity of field names is typically employed. This method calculates the edit distance, string similarity, or word vector similarity between field names, identifying field pairs with similarity exceeding a preset fixed threshold as matches, and establishing field mapping relationships accordingly. This method is simple to implement, computationally inexpensive, and has certain applicability in homogeneous data source scenarios with standardized field naming conventions.
[0004] However, in multi-source heterogeneous data environments, each data source is independently constructed with different naming systems. Semantically equivalent fields often have significant differences in name. For example, "ID number," "patient_id," and "last four digits of social security card number" are actually the same type of identification information. Methods that rely solely on the literal similarity of field names cannot identify such semantic equivalence relationships, resulting in a persistently high false match rate. At the same time, existing methods use a uniform fixed similarity threshold for all fields, failing to consider the differences in the business importance of fields in the data fusion task. This makes it difficult to reduce the false match rate of key fields while ensuring the overall matching recall rate, leading to inaccurate field matching results. Summary of the Invention
[0005] To address the technical problem of inaccurate field matching results, this application provides an AI multi-source data processing method and system based on big data. This method can impose stricter matching constraints on high-weight fields and maintain a loose threshold for low-weight fields. While reducing the false matching rate of key fields, it maintains the overall matching recall rate, thus achieving synergistic optimization of matching precision and recall.
[0006] In a first aspect, this application provides an AI multi-source data processing method based on big data. The method includes: collecting field names, total number of records, number of non-empty records, and content samples of each field from multiple heterogeneous data sources; and statistically analyzing the frequency of each field pair appearing as a common association key from historical data fusion task records to construct a field co-occurrence matrix; calculating the field correlation degree based on the co-occurrence frequency of each field with all other fields in the field co-occurrence matrix; using the ratio of the number of non-empty records to the total number of records as data completeness; and using the product of the field correlation degree and data completeness as the field weight of each field; and performing multiple random operations on each field. Extract content samples, obtain the semantic vector of each content sample, and use the mean of the semantic vector as the centroid vector; calculate the mean cosine distance between each semantic vector and the centroid vector, use the complement of the mean cosine distance as the aggregation stability, and calculate the cosine similarity of the centroid vectors between any two fields from different data sources as the cross-source similarity; construct an adaptive similarity threshold for each field, which is positively correlated with the field weight and negatively correlated with the aggregation stability; for each pair of cross-source fields, if the cross-source similarity is not lower than the adaptive similarity threshold, then the field pair is determined to be semantically matched, and the field mapping relationship is output.
[0007] By generating centroid vectors through multiple random samplings and calculating aggregation stability, cross-source similarity is calculated based on the noise-resistant centroid vectors. Finally, adaptive similarity thresholds for each field are determined by field weights and aggregation stability, enabling differentiated matching decisions for different fields. This effectively reduces the false matching rate and improves the reliability of field alignment in scenarios with high field heterogeneity.
[0008] Preferably, the step of constructing the field co-occurrence matrix includes: extracting historical association operation records within a preset time window from historical data fusion task records, counting the number of times each pair of fields appears as an association key in the JOIN operation; constructing a symmetric matrix with the total number of all fields as the dimension, filling the co-occurrence count of each pair of fields into the corresponding matrix elements, and obtaining the field co-occurrence matrix.
[0009] Preferably, the step of calculating the field correlation degree includes: dividing the sum of the number of times any field co-occurs with all other fields in the field co-occurrence matrix by the maximum value of the sum of the number of times all fields co-occur, to obtain the field correlation degree; the value of the field correlation degree is between zero and one.
[0010] Preferably, the step of randomly sampling the content samples multiple times to obtain the semantic vector corresponding to each content sample includes: performing K independent random samplings for each field, randomly selecting M non-empty records from the content samples each time, concatenating the M non-empty records with the field name to form input text, inputting it into a pre-trained language model to obtain the semantic vector of that independent random sampling, and obtaining K semantic vectors.
[0011] By performing K independent random samplings on each field, and selecting M non-empty records each time and concatenating them with the field name before inputting them into a pre-trained language model, multiple sets of semantic vectors are generated. This makes the semantic representation generation process independent of the single fixed selection result of the content sample, thereby enabling the semantic vectors to more comprehensively reflect the true semantics of the field in the business scenario.
[0012] Preferably, the pre-trained language model is a BERT model, an LSTM model, or a Transformer model.
[0013] Preferably, in the step of constructing the adaptive similarity threshold for each field, the adaptive similarity threshold is also negatively correlated with the matching significance.
[0014] When a field has multiple candidate matching objects with similar cross-source similarity, the matching significance is low. The adaptive similarity threshold is increased accordingly to filter out erroneous candidate objects with the second highest similarity, reduce the probability of ambiguous matching, and enable the adaptive similarity threshold to perceive the local uncertainty of the current matching task, thereby improving the adaptability of the matching decision to complex matching scenarios.
[0015] Preferably, the step of obtaining the matching saliency includes: for each field, sorting its cross-source similarity with all fields in the target data source in descending order, and subtracting the second largest cross-source similarity from the largest cross-source similarity to obtain the matching saliency.
[0016] Preferably, the step of constructing the adaptive similarity threshold for each field includes: using the product of the field weight, the complement of the matching saliency, and the first adjustment coefficient as a first adjustment term; using the product of the field weight, the complement of the aggregation stability, and the second adjustment coefficient as a second adjustment term; and using the sum of the preset baseline threshold, the first adjustment term, and the second adjustment term as the adaptive similarity threshold for the corresponding field.
[0017] By clearly distinguishing the contribution of matching uncertainty risk (first adjustment term) and semantic representation instability risk (second adjustment term) to the threshold through two adjustment terms, high-weight fields are subject to the superimposed constraints from both types of risks, while low-weight fields, even if they face higher matching uncertainty or semantic instability, avoid a large number of missed matches caused by setting excessively high thresholds for secondary fields.
[0018] Preferably, the processing method further includes: comparing the aggregation stability with a preset stability threshold; if the aggregation stability is lower than the preset stability threshold, adding a semantically unstable marker to the field; and when outputting field mapping relationships, adding the semantically unstable marker to the field mapping relationships corresponding to the fields carrying the semantically unstable marker.
[0019] Add a semantically unstable marker to fields with unstable semantic representations, and pass the marker along with the mapping result when outputting field mapping relationships. Operators can directly review the relevant mapping relationships based on the semantically unstable marker, which improves the overall level of automation and provides effective assurance for the quality control of data fusion.
[0020] In a second aspect, this application also provides an AI multi-source data processing system based on big data, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the AI multi-source data processing method based on big data according to the first aspect of this application is implemented.
[0021] The technical solution of this application has the following beneficial technical effects:
[0022] By integrating three types of information—historical usage patterns, data quality characteristics, and semantic content characteristics—this application constructs a complete process from field importance quantification, semantic representation generation, semantic reliability assessment, to differentiated matching. Compared to existing technologies that rely solely on the literal similarity of field names for field matching, this application can still accurately identify semantically equivalent fields and obtain accurate field matching results even in scenarios with highly heterogeneous field naming systems. Furthermore, by dynamically coupling the business importance of a field with matching uncertainty into an adaptive similarity threshold, stricter matching constraints are imposed on high-weight fields, while a more lenient threshold is maintained for low-weight fields. This reduces the false matching rate of key fields without sacrificing the overall matching recall rate, achieving synergistic optimization of matching precision and recall. Attached Figure Description
[0023] Figure 1 This is a flowchart of an AI multi-source data processing method based on big data according to an embodiment of this application.
[0024] Figure 2 This is a performance comparison chart of the method of this application according to embodiments of this application versus field similarity-based methods.
[0025] Figure 3 This is a structural block diagram of an AI multi-source data processing system based on big data, according to an embodiment of this application. Detailed Implementation
[0026] According to the first aspect of this application, this application provides an AI-based multi-source data processing method based on big data. This embodiment uses a multi-source field mapping task of a municipal government data platform as an application scenario for illustration. This municipal government data platform needs to perform cross-source association queries on citizen data scattered in the public security population information system (relational database), medical and health record system (document database), social insurance ledger (CSV file), and administrative approval log system (log system). Each system is independently constructed, and the field naming system is different. For example, the field name storing citizen identity information in the public security system is "ID number", the corresponding field name in the medical system is "patient_id", and the corresponding field name in the social security system is "last four digits of social security card number association code". The accuracy rate of matching directly by field name is less than 30%, which cannot meet the needs of data fusion. To this end, this application provides an AI-based multi-source data processing method based on big data, which automatically establishes the mapping relationship between fields of each data source by combining the historical usage pattern of fields, data quality characteristics, and semantic content characteristics, providing accurate pattern alignment results for subsequent data fusion and cross-source queries.
[0027] Figure 1 This is a flowchart of an AI multi-source data processing method based on big data, according to an embodiment of this application. Figure 1 As shown, the AI multi-source data processing method based on big data includes steps S101 to S105, which are described in detail below.
[0028] S101 collects the field name, total number of records, number of non-empty records, and content samples of each field from multiple heterogeneous data sources, and counts the number of times each field pair appears as a common association key from historical data fusion task records to construct a field co-occurrence matrix.
[0029] In one embodiment, for the aforementioned government data platform scenario, metadata for each field is first extracted from four heterogeneous data sources: the public security population information system, the medical and health record system, the social insurance ledger, and the administrative approval log system. This metadata serves as the data foundation for subsequent field importance assessment and semantic matching. For each field, the following four types of information are collected: field name (to identify the field); total number of records (the total number of rows in the data table containing the field); number of non-empty records (the number of rows with actual values in the field); and content samples (a number of non-empty records randomly selected from the field for subsequent semantic vector generation).
[0030] For example, in the public security population information system, the field name of the field "ID number" is "ID number", the total number of records is 1,200,000, the number of non-empty records is 1,198,000, and the content sample is 100 randomly selected non-empty ID number records.
[0031] The construction of the field co-occurrence matrix aims to quantify the historical association usage patterns between fields, so as to evaluate the pivotal role of each field in the data fusion task. The specific construction method of the field co-occurrence matrix is as follows: extract historical association operation records within a preset time window from the historical data fusion task records, and count the number of times each pair of fields appears as a common association key in the JOIN operation; construct a symmetric matrix with the total number of all fields as the dimension, and fill the corresponding matrix elements with the co-occurrence frequency of each field pair to obtain the field co-occurrence matrix.
[0032] It should be noted that the preset time window refers to the time range used to extract historical related operation records, and its value can be configured according to the data accumulation cycle of the actual business. The preferred value is the past 30 days. This value can capture a sufficient number of historical related operation samples while avoiding the introduction of overly outdated related patterns, ensuring the accuracy of the field co-occurrence matrix in reflecting the current business needs.
[0033] The field co-occurrence matrix is a A symmetric matrix, The total number of fields. The matrix number... Line number The elements of a column represent fields. With fields The number of times they appear together as a join key in historical data fusion tasks. In a government data platform scenario, if four data sources cover a total of 120 fields, the field co-occurrence matrix is a symmetric matrix of 120×120. For example, if "ID number" and "patient_id" appeared together as a join key 87 times in the past 30 days of historical fusion tasks, the corresponding element in the matrix would have a value of 87.
[0034] In this way, by collecting the field name, total number of records, number of non-empty records, and content samples of each field, and constructing a field co-occurrence matrix based on historical associated operation records, the abstract field usage rules are represented in a structured matrix form, laying a data foundation for subsequent quantitative evaluation of the fields.
[0035] S102, calculate the field correlation degree based on the number of times each field co-occurs with all other fields in the field co-occurrence matrix, use the ratio of the number of non-empty records to the total number of records as the data completeness, and use the product of the field correlation degree and the data completeness as the field weight.
[0036] In one embodiment, based on the field co-occurrence matrix, the total number of records for each field, and the number of non-empty records, the field correlation degree and data completeness are calculated respectively, and then the field weights are obtained to comprehensively characterize the importance of each field in the data fusion task.
[0037] Field correlation reflects the frequency with which a field acts as a correlation key in historical data fusion tasks. A higher value indicates that the field is a crucial node in data fusion, and a mismatch will have a greater impact on the overall fusion result. The specific calculation method for field correlation is as follows: the sum of the co-occurrence counts of any field with all other fields in the field co-occurrence matrix is divided by the maximum sum of the co-occurrence counts of all fields. The field correlation value ranges from zero to one.
[0038] The field correlation satisfies the following relational formula:
[0039]
[0040] In the formula, For fields The field correlation, with a value range of 100%. ; Total number of fields; For fields With fields The co-occurrence count is used as the denominator, which is the maximum sum of co-occurrence counts across all fields, and serves as a normalization factor. If the denominator is zero, the field correlation is assigned a value of zero.
[0041] Understandably, the above formula is based on the principle of degree centrality in graph theory, measuring the structural importance of a field by the strength of its connections in the network. The more times a field co-occurs with other fields in history, the stronger its bridging role in cross-source data fusion, the higher its field correlation, and the greater its importance for accurate matching. In the aforementioned government data platform scenario, the "ID number" field has a correlation close to 1 because it frequently appears in JOIN operations of various cross-system fusion tasks; the "remarks" field has a correlation close to 0 because it almost never participates in JOIN operations.
[0042] Data completeness reflects the quality of data population in a field. A higher value indicates a higher degree of standardized data entry in business operations, and a more reliable semantic vector generated based on that field's content. Data completeness satisfies the following relation:
[0043]
[0044] In the formula, For fields Data integrity, with a value range of [value missing]. ; For fields The number of non-empty records; For fields The total number of records; when the total number of records is 0, the data integrity value is set to 0.
[0045] Understandably, this formula conforms to the definition of the integrity dimension in the ISO / IEC 25012 data quality model, measuring the usability of field data by the proportion of non-empty records to the total number of records. In the above scenario, the number of non-empty records in the "ID Number" field is 1,198,000, and the total number of records is 1,200,000, resulting in a data integrity score of 0.998; the number of non-empty records in the "Remarks" field is only 240,000, and the total number of records is 1,200,000, resulting in a data integrity score of 0.20. This indicates that the data filling standardization of the "Remarks" field is poor, affecting the representativeness of its content sample.
[0046] Field weights combine field relevance and data completeness to ultimately measure the overall importance of a field in the overall data fusion task. Field weights satisfy the following relational expression:
[0047]
[0048] In the formula, For fields The field weight, with a value range of 100%. ; For field correlation; For data integrity.
[0049] Understandably, using multiplication instead of addition to fuse the two dimensions reflects their synergistic and restrictive relationship. When a field has high correlation but low data completeness, its weight will be significantly reduced, preventing misleading high-confidence matching of important fields due to data quality defects. For example, the field weight of "ID number" is... The field weight of the "Remarks" field is... The former will be subject to stricter matching constraints in subsequent adaptive similarity threshold adjustments.
[0050] In this way, by extracting field relevance and data completeness from the two dimensions of structural importance and data quality respectively, and obtaining field weights by multiplication, a comprehensive characterization of the business value of fields is achieved, providing a reliable quantitative basis for applying stricter matching constraints to high-weight fields in the future.
[0051] S103, randomly sample the content multiple times for each field, obtain the semantic vector corresponding to each content sample, and use the mean of the semantic vector as the centroid vector; calculate the average cosine distance between each semantic vector and the centroid vector, use the complement of the average cosine distance as the aggregation stability, and calculate the cosine similarity of the centroid vectors between any two fields from different data sources as the cross-source similarity.
[0052] In one embodiment, to obtain a reliable semantic representation of a field, a semantic vector is generated by multiple random samplings for each field, and the mean of the multiple sampling vectors is used as the centroid vector of the field, thereby eliminating the semantic bias caused by random factors and noisy data in a single sampling.
[0053] The steps for obtaining the semantic vector corresponding to each content sample by randomly sampling the content sample multiple times are as follows: perform K independent random samplings for each field, randomly select M non-empty records from the content sample each time, concatenate the M non-empty records with the field name to form the input text, input it into the pre-trained language model, and obtain the semantic vector of the independent random sampling, thus obtaining K semantic vectors.
[0054] It should be noted that K is the number of samplings, a key hyperparameter controlling the stability of the semantic vector. In this embodiment, K=5 is preferred, meaning that each field is sampled 5 times independently and randomly. M is the number of non-empty records sampled each time. In this embodiment, M=20 is preferred, meaning that 20 non-empty records are randomly selected from the content sample each time. The M non-empty records are concatenated with the field names and input into the pre-trained language model to obtain the semantic vector of that sampling.
[0055] The pre-trained language model can be any of the BERT, LSTM, or Transformer models. Understandably, the BERT model, pre-trained on a large-scale corpus, is capable of deep semantic understanding of the combined input of field names and content samples, capturing the actual semantic meaning of the fields in the business scenario.
[0056] After obtaining K semantic vectors, the centroid vector is calculated. The centroid vector is obtained by averaging the semantic vectors from multiple samplings, resulting in a stable semantic representation that integrates information from multiple samplings. This effectively smooths out the semantic vector bias caused by randomly sampling noisy data or abnormal records in a single sampling, making the results of subsequent cross-source similarity calculations more stable and reliable.
[0057] After obtaining the centroid vector, the mean cosine distance between each semantic vector and the centroid vector is further calculated as an indicator to measure the dispersion of semantic vectors sampled multiple times, and the complement of the mean cosine distance is used as the aggregation stability.
[0058] The cosine distance is defined using the standard cosine distance, which is the difference between 1 and the cosine similarity. Aggregation stability satisfies the following relationship:
[0059]
[0060] In the formula, For fields The polymerization stability, in practice, typically takes the value of... Within the range; For fields The average cosine distance.
[0061] Understandably, aggregation stability is defined by the complement of the mean cosine distance, which positively correlates it with the concentration of semantic vectors. When semantic vectors from multiple samplings are highly concentrated near the centroid vector, the mean cosine distance approaches 0, and the aggregation stability approaches 1. This indicates that the field has clear semantics and standardized data, and the centroid vector can reliably represent the true semantics of the field. Conversely, when semantic vectors are dispersed, aggregation stability is low, indicating that the field has high content diversity or semantic ambiguity, and the representativeness of the centroid vector is affected. For example, the "ID number" field, because its content is all in the standard ID card number format, has highly similar semantic vectors across different samples, resulting in an aggregation stability close to 0.95. The "remarks" field, because its content covers various types of free text, has significant semantic differences across different samples, and its aggregation stability may be as low as 0.45.
[0062] Subsequently, the cosine similarity of the centroid vectors between any two fields from different data sources is calculated as the cross-source similarity. It should be noted that using centroid vectors instead of semantic vectors from a single sampling for cross-source similarity calculation can effectively avoid the risk of bias in similarity calculation caused by randomly sampling noisy records in a single sampling, making the cross-source similarity calculation results more stable and reliable.
[0063] S104, construct an adaptive similarity threshold for each field, wherein the adaptive similarity threshold is positively correlated with the field weight and negatively correlated with the aggregation stability.
[0064] In one embodiment, a higher field weight indicates greater importance of the field in data fusion. False matching will significantly impact the overall fusion result, thus requiring a higher matching threshold to mitigate the risk of false matching. Conversely, lower aggregation stability indicates lower reliability of the field's semantic representation, with uncertainty in the representativeness of the centroid vector. Similarly, increasing the adaptive similarity threshold is necessary to avoid false matching risks caused by semantic representation errors. In summary, the adaptive similarity threshold is positively correlated with field weight and negatively correlated with aggregation stability; it is the sum of the preset baseline threshold and the second adjustment term, which is the product of the field weight, the complement of aggregation stability, and the first adjustment coefficient.
[0065] Based on the correlation between the adaptive similarity threshold and field weights and aggregation stability, the adaptive similarity threshold is also negatively correlated with the matching significance. It should be noted that the lower the matching significance, the more likely there are multiple candidate matching objects with similar cross-source similarity, and the lower the uniqueness of the matching result. Therefore, the adaptive similarity threshold needs to be increased to exclude erroneous candidate objects with the second highest similarity. On the other hand, the higher the matching significance, the more likely the best matching target has been identified and is highly distinguishable from other candidate objects. The threshold can be appropriately relaxed to avoid missed matches.
[0066] The specific steps for obtaining match saliency are as follows: For each field, sort its cross-source similarity with all fields in the target data source in descending order, and subtract the second largest cross-source similarity from the largest cross-source similarity to obtain the match saliency. The match saliency satisfies the following relationship:
[0067]
[0068] In the formula, For fields The saliency of the match, the value range is usually in the range of Within the range; For fields The maximum cross-source similarity among all fields in the target data source; It is the second largest cross-source similarity value after descending order.
[0069] Understandably, if the maximum cross-source similarity is much greater than the second largest cross-source similarity, it means that the best matching target for this field is unique and clear, and the matching result is reliable. If the two are close, it means that there are multiple candidate objects with similar cross-source similarity for this field, and the matching result has high uncertainty. More stringent constraints need to be applied in the adaptive similarity threshold adjustment.
[0070] In one embodiment, the product of the field weight, the complement of the matching saliency, and the first adjustment coefficient is used as the first adjustment term; the product of the field weight, the complement of the aggregation stability, and the second adjustment coefficient is used as the second adjustment term; and the sum of the preset baseline threshold, the first adjustment term, and the second adjustment term is used as the adaptive similarity threshold of the corresponding field.
[0071] The adaptive similarity threshold satisfies the following relationship:
[0072]
[0073] In the formula, For fields The adaptive similarity threshold, with a value range of [value range missing]. ; The preset baseline threshold is obtained by statistically analyzing the cross-source similarity distribution of correctly matched field pairs on historical datasets, using the lowest similarity value that makes the mismatch rate lower than an acceptable level as a reference. In this embodiment, the value is 0.75. The first adjustment coefficient controls the magnitude of the adjustment of the adaptive similarity threshold by the matching uncertainty. It is obtained by applying different coefficients to the validation set. Experiments were conducted to select the value that significantly improves matching precision while minimizing recall loss; the preferred value was 0.15. The second adjustment coefficient controls the magnitude of semantic instability's influence on the adaptive similarity threshold; its acquisition method is the same as... The preferred value is 0.10; Assign field weights; To match saliency; For polymerization stability.
[0074] Understandably, the adaptive similarity threshold includes two adjustment terms: the first adjustment term... This reflects the risk of important fields having uncertain matches. When the field weight is high and the match significance is low, the first adjustment term is larger, and the adaptive similarity threshold is correspondingly increased, requiring higher cross-source similarity to trigger a match determination; the second adjustment term... This reflects the risk of important fields with unstable semantics. When the field weight is high and the aggregation stability is low, the second adjustment term is larger, further increasing the adaptive similarity threshold and applying additional matching protection to important fields with uncertain semantic representations. The sum of the two adjustment coefficients is 0.25, ensuring that the maximum adjustment of the adaptive similarity threshold is 0.25. This strictly constrains the matching of core fields while avoiding the possibility of a large number of correct matches being incorrectly filtered out due to an excessively high threshold.
[0075] For example, for a field with a weight of 0.9, a matching significance of 0.4, and an aggregation stability of 0.6, its adaptive similarity threshold is: The adaptive similarity threshold is significantly higher than the preset baseline threshold of 0.75, ensuring that only field pairs with extremely high cross-source similarity can be judged as matches, effectively reducing the probability of false matches for this important field.
[0076] Thus, by combining field weights, matching saliency, and aggregation stability, an adaptive similarity threshold is constructed for each field. Compared to the fixed threshold method, the adaptive similarity threshold imposes stricter matching constraints on high-weight, high-uncertainty fields, significantly reducing the mismatch rate at key nodes of data fusion.
[0077] S105, for each pair of cross-source fields, if the cross-source similarity is not lower than the adaptive similarity threshold, then the pair of fields is determined to be semantically matched, and the field mapping relationship is output.
[0078] In this embodiment, matching is performed on all cross-source field pairs based on cross-source similarity and an adaptive similarity threshold. For each field in data source A... Iterate through all fields in data source B. If the field With fields Cross-source similarity Not lower than the field Adaptive similarity threshold Then determine the field With fields For semantic matching, record the mapping relationship of the field; traverse all cross-source field pairs and output the mapping relationship of all matching fields for subsequent data fusion operations.
[0079] During the process of outputting field mapping relationships, the aggregation stability is compared with a preset stability threshold. If the aggregation stability is lower than the preset stability threshold, a semantically unstable flag is added to the field. When outputting field mapping relationships, the semantically unstable flag is added to the field mapping relationship corresponding to the field carrying the semantically unstable flag.
[0080] It should be noted that the preset stability threshold is a configuration parameter that controls the sensitivity of semantic instability warnings. It is obtained by statistically analyzing the aggregation stability distribution of different types of fields on a real dataset, using the critical value that distinguishes semantically explicit fields from semantically ambiguous fields as a reference. A preferred value is 0.7. That is, when the aggregation stability is below 0.7, a semantically unstable flag is added to the field, prompting subsequent data fusion operators to manually review the matching results for that field, further reducing the spread of mismatch risks caused by semantic instability during data fusion.
[0081] For example, in the above-mentioned government data platform scenario, if the cross-source similarity between "ID number" and "patient_id" is 0.91 and the adaptive similarity threshold of "ID number" is 0.867, then 0.91 ≥ 0.867, and the two are determined to be semantically matched, and the field mapping relationship "ID number → patient_id" is output; if the aggregation stability of the "remarks" field is 0.45, which is lower than the preset stability threshold of 0.7, then the "remarks" field carries a semantically unstable marker, and the corresponding field mapping relationship entry is appended with a semantically unstable marker when output, prompting the operator to manually review the mapping relationship.
[0082] Please see Figure 2 The graph shows a performance comparison between the proposed method and field similarity-based methods according to embodiments of this application. The proposed method significantly improves matching precision and recall to 83.5% and 82%, respectively, while the false match rate is drastically reduced from 21.5% to 14.8%. These results clearly demonstrate that the proposed method reduces the false match rate of key fields without sacrificing overall matching recall, achieving synergistic optimization of matching precision and recall.
[0083] In this way, while outputting the field mapping relationship, it provides clear risk warnings for field mapping results with uncertain semantic representation, improves the reliability of multi-source data fusion results, and achieves synergistic optimization of matching precision and recall.
[0084] According to a second aspect of this application, this application also provides an AI multi-source data processing system based on big data. Figure 3 This is a structural block diagram of an AI multi-source data processing system based on big data, according to an embodiment of this application. Figure 3 As shown, the system 50 includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the AI multi-source data processing method based on big data according to the first aspect of this application. The system also includes other components well-known to those skilled in the art, such as a communication bus and a communication interface. Their configuration and functions are known in the art and will not be described further here.
[0085] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the scope of protection of this application.
Claims
1. A multi-source AI data processing method based on big data, characterized in that, The processing method includes: collecting field names, total number of records, number of non-empty records, and content samples of each field from multiple heterogeneous data sources; and counting the number of times each field pair appears as a common association key from historical data fusion task records, constructing a field co-occurrence matrix, including: Extract historical related operation records within a preset time window from historical data fusion task records, and count the number of times each pair of fields appears as a related key in JOIN operations; A symmetric matrix is constructed with the total number of all fields as the dimension, and the co-occurrence frequency of each field pair is filled into the corresponding matrix elements to obtain the field co-occurrence matrix; The field correlation degree is calculated based on the number of times each field co-occurs with all other fields in the field co-occurrence matrix. The ratio of the number of non-empty records to the total number of records is used as the data completeness. The product of the field correlation degree and the data completeness degree is used as the field weight of each field. For each field, content samples are randomly drawn multiple times to obtain the semantic vector of each content sample and the mean of the semantic vector is used as the centroid vector; the mean cosine distance between each semantic vector and the centroid vector is calculated, the complement of the mean cosine distance is used as the aggregation stability, and the cosine similarity of the centroid vectors between any two fields from different data sources is calculated as the cross-source similarity. An adaptive similarity threshold is constructed for each field. The adaptive similarity threshold is positively correlated with the field weight, negatively correlated with the aggregation stability, and also negatively correlated with the matching saliency. For each pair of cross-source fields, if the cross-source similarity is not lower than the adaptive similarity threshold, then the pair of fields is determined to be semantically matched, and the field mapping relationship is output.
2. The AI multi-source data processing method based on big data according to claim 1, characterized in that, The steps for calculating the correlation degree of the fields include: The field correlation degree is obtained by dividing the sum of the co-occurrence counts of any field with all other fields in the field co-occurrence matrix by the maximum sum of the co-occurrence counts of all fields. The value of the correlation degree of the field ranges from zero to one.
3. The AI multi-source data processing method based on big data according to claim 1, characterized in that, The step of randomly sampling the content samples multiple times to obtain the semantic vector corresponding to each content sample includes: performing K independent random samplings for each field, randomly selecting M non-empty records from the content samples each time, concatenating the M non-empty records with the field name to form input text, inputting it into a pre-trained language model to obtain the semantic vector of that independent random sampling, and obtaining K semantic vectors.
4. The AI multi-source data processing method based on big data according to claim 3, characterized in that, The pre-trained language model is a BERT model, an LSTM model, or a Transformer model.
5. The AI multi-source data processing method based on big data according to claim 1, characterized in that, The steps for obtaining the matching saliency include: for each field, sorting its cross-source similarity with all fields in the target data source in descending order, and subtracting the second largest cross-source similarity from the largest cross-source similarity to obtain the matching saliency.
6. The AI multi-source data processing method based on big data according to claim 1, characterized in that, The steps for constructing the adaptive similarity thresholds for each field include: The product of the field weight, the complement of the matching saliency, and the first adjustment coefficient is used as the first adjustment term; the product of the field weight, the complement of the aggregation stability, and the second adjustment coefficient is used as the second adjustment term; and the sum of the preset benchmark threshold, the first adjustment term, and the second adjustment term is used as the adaptive similarity threshold of the corresponding field.
7. The AI multi-source data processing method based on big data according to claim 1, characterized in that, The processing method further includes: The aggregation stability is compared with a preset stability threshold. If the aggregation stability is lower than the preset stability threshold, a semantically unstable flag is added to the field. When outputting field mapping relationships, the semantically unstable flag is added to the field mapping relationship corresponding to the field carrying the semantically unstable flag.
8. A multi-source AI data processing system based on big data, characterized in that, It includes a processor and a memory, the memory storing computer program instructions, which, when executed by the processor, implement the AI multi-source data processing method based on big data according to any one of claims 1 to 7.