Data comparison method based on AI
By using AI technology for data preprocessing, text vectorization, and machine learning models, the shortcomings of traditional data comparison methods in semantic understanding and format processing are addressed. This enables efficient and flexible data comparison, adapts to multi-source heterogeneous data, reduces false positive and false negative rates, and improves system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-16
- Publication Date
- 2026-03-24
AI Technical Summary
Existing data comparison technologies are ill-equipped to handle semantic differences, inconsistent formats, and unstructured data. They are inflexible, have high false positive and false negative rates, poor scalability, cannot handle synonyms and format differences, and lack intelligent judgment capabilities.
We employ AI methods for data preprocessing, text vectorization, similarity calculation and candidate pair generation, matching decision, and post-processing evaluation. By cleaning and standardizing the text, we use models such as Word2Vec/GloVe/BERT for semantic understanding, combine machine learning classifiers for matching decisions, and optimize the model through feedback loops.
It improves tolerance to format differences and noise, supports unstructured data comparison, reduces false positive and false negative rates, has self-learning capabilities, adapts to dynamic changes in data structure, and improves the accuracy and automation level of comparison.
Smart Images
Figure CN121723992A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing, and more particularly to an AI-based data comparison method. Background Technology
[0002] Against the backdrop of accelerated digital transformation, data has become a core asset. However, the diverse sources, heterogeneous formats, and frequent updates of data pose significant challenges to data consistency, accuracy, and integrity. Traditional data comparison methods based on rule-based or field-specific precise matching struggle to address issues such as semantic differences, inconsistent formats, and unstructured data, necessitating more intelligent technological approaches. With the development of artificial intelligence, natural language processing, and machine learning, AI-based data comparison technologies have emerged, enabling semantic understanding, fuzzy matching, automatic alignment, and anomaly detection. These technologies significantly improve the accuracy and automation of comparisons, finding widespread application in scenarios such as data migration, system integration, quality monitoring, and compliance auditing, becoming a crucial support for ensuring data quality and trusted computing.
[0003] Existing technical solutions: Rule-based precise comparison: By using preset field mapping relationships and matching rules (such as equal primary keys, consistent field names), structured data is compared field by field. It is suitable for database tables or Excel files with uniform format and identical structure. Common tools include SQL queries and Excel functions (such as VLOOKUP).
[0004] Fuzzy matching and similarity calculation: To address issues such as spelling errors and naming differences, algorithms such as edit distance, cosine similarity, and Jaccard coefficient are used to determine the degree of similarity between data. This is often used for comparing text fields such as customer names and addresses.
[0005] Hash comparison and deduplication: Generate unique fingerprints for data using hash algorithms such as MD5 and SHA to quickly identify duplicate or different records. Suitable for efficient comparison of large amounts of data.
[0006] ETL tools and data integration platforms: Utilizing tools such as Kettle, DataStage, and Informatica, comparison and verification are achieved during data extraction and transformation, supporting the integration of heterogeneous data sources and automated processes.
[0007] Visual comparison tools, such as Beyond Compare and WinMerge, provide graphical interfaces to intuitively display the differences between files or databases, facilitating manual review.
[0008] Existing data comparison technologies have the following main drawbacks: Relying on manual rules and lacking flexibility: Traditional methods require pre-defining field mapping and matching rules, making it difficult to cope with structural changes or new data sources, resulting in high maintenance costs.
[0009] Unable to handle semantic differences: It has a weak ability to recognize synonyms and inconsistent naming (such as "customer name" vs "user name"), and is prone to missed matches or misjudgments.
[0010] Sensitive to format differences: Inconsistent formats for dates, amounts, etc. (e.g., “2023-01-01” vs “01 / 01 / 2023”) will lead to comparison failures and require extensive preprocessing.
[0011] It struggles with unstructured data: unstructured content such as text, logs, and PDFs cannot be effectively compared.
[0012] High false alarm and false negative rates: fuzzy matching thresholds are difficult to set and are prone to generating a lot of noise; dynamic or random fields (such as timestamps) interfere with the comparison results.
[0013] Poor scalability: When faced with large-scale, multi-source heterogeneous data, performance degrades significantly, making it difficult to achieve automated and real-time comparison.
[0014] Lack of intelligent judgment: unable to assess the business importance of differences, treating all changes "the same", increasing the burden of manual review. Summary of the Invention
[0015] In view of the above problems, the present invention is proposed to provide an AI-based data comparison method that overcomes or at least partially solves the above problems.
[0016] According to one aspect of the present invention, an AI-based data comparison method is provided, the data comparison method comprising: Data preprocessing involves cleaning and standardizing the raw text into regular text. Text vectorization processing converts the regularized text into a numerical form that machines can understand and calculate; Similarity calculation and candidate pair generation; Matching decision: Based on the calculated similarity score, determine whether two texts refer to the same entity and obtain the matching result; Post-processing and evaluation involve optimizing the matching results and scientifically evaluating the performance of the entire system.
[0017] Optionally, the data preprocessing, which cleans and standardizes the original text into regularized text, specifically includes: Cleaning: Remove irrelevant characters: Remove HTML tags, special characters, extra spaces, and line breaks; Handling missing values: Filling in or deleting fields with missing text; Standardization: Case unification: Convert all characters to lowercase; Abbreviations and abbreviation extensions: restoring common abbreviations to their full forms; Standardized format: Standardize the format of dates, numbers, and units; Word segmentation: For English, word segmentation by spaces and punctuation is relatively simple; For Chinese, word segmentation tools are needed to break down continuous Chinese text into individual words; Remove stop words: Remove frequently occurring words that contain little information to highlight key information; Stem extraction and word form reduction: Reducing words to their root form.
[0018] Optionally, the text vectorization process, which converts the regularized text into a numerical form that a machine can understand and calculate, specifically includes: Word2Vec / GloVe is used to map words to vectors, with semantically similar words being closer together in the vector space; an entire sentence or phrase is mapped to a fixed-length vector.
[0019] Optionally, the similarity calculation and candidate pair generation specifically includes: calculating the similarity score between text pairs and filtering out the candidate pairs that are most likely to match.
[0020] Optionally, calculating the similarity score between text pairs and filtering out the most likely matching candidate pairs specifically includes: Blocking: Grouping data using a coarse-grained key and comparing only within the group; Commonly used blocking keys: first letter of Pinyin, postal code, industry category, keywords; Similarity calculation: For two texts in a candidate pair, calculate the similarity between their vector representations; use cosine similarity to measure the directional consistency of the two vectors, the result is between [-1, 1], and the value is between [0, 1], the larger the value, the more similar they are.
[0021] Optionally, the matching decision, based on the calculated similarity score, determines whether two texts point to the same entity, and the specific steps to obtain the matching result include: If the score is higher than the threshold, it is considered a match; if it is lower, it is considered a non-match. Prepare a labeled training set using an AI machine learning classifier; Input the features of the text pairs into the classification model for training; Commonly used models include: gradient boosting trees, logistic regression, support vector machines, or deep Siamese networks; The model outputs a matching probability and makes a final decision based on the set probability threshold.
[0022] Optionally, the features of the text pair specifically include: similarity scores of multiple fields and the vector itself.
[0023] Optionally, the post-processing and evaluation, which optimizes the matching results and scientifically evaluates the performance of the entire system, specifically includes: Evaluation indicators include: Precision: Of the candidate pairs that the system judges as "matches", how many are actually matches; Recall: Of all truly matching candidate pairs, the number successfully identified by the system. F1-Score: The harmonic mean of precision and recall, which is a comprehensive metric. Feedback loop: Add cases where the system misjudged to the training set, retrain the model, and continuously iterate and optimize.
[0024] This invention provides an AI-based data comparison method, comprising: data preprocessing, which cleans and standardizes the original text into regularized text; text vectorization, which converts the regularized text into a numerical form that can be understood and calculated by a machine; similarity calculation and candidate pair generation; matching decision, which determines whether two texts refer to the same entity based on the calculated similarity score, and obtains a matching result; and post-processing and evaluation, which optimizes the matching result and scientifically evaluates the performance of the entire system. This method improves tolerance to format differences and noise, and achieves automation and scalability.
[0025] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 A flowchart of an AI-based data comparison method provided in an embodiment of the present invention. Detailed Implementation
[0028] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0029] The terms "comprising" and "having," and any variations thereof, in the specification, embodiments, claims, and drawings of this invention are intended to cover non-exclusive inclusion, such as including a series of steps or units.
[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.
[0031] like Figure 1 As shown, an AI-based data comparison method includes: Data preprocessing involves cleaning and standardizing the raw text into regular text. Text vectorization processing converts the regularized text into a numerical form that machines can understand and calculate; Similarity calculation and candidate pair generation; Matching decision: Based on the calculated similarity score, determine whether two texts refer to the same entity and obtain the matching result; Post-processing and evaluation involve optimizing the matching results and scientifically evaluating the performance of the entire system.
[0032] Step 1: Data Preprocessing Objective: To clean and standardize the messy raw text into relatively regular text, laying a solid foundation for subsequent AI model processing. Ensure that data is processed in a "garbage-in, garbage-out" manner.
[0033] Main operations: Cleaning: Remove irrelevant characters: Remove HTML tags, special characters, extra spaces, and line breaks.
[0034] Handling missing values: Filling in or deleting fields with missing text.
[0035] Standardization: Case uniformity: Generally, all characters are converted to lowercase to ensure that "Apple" and "apple" are treated equally.
[0036] Abbreviations and abbreviations expansion: Restore common abbreviations to their full names. For example: "Co." -> "Company", "Ltd." -> "Limited" Standardize the format: Standardize the format of dates, numbers, units, etc.
[0037] Word Segmentation: For English, segmenting by spaces and punctuation is relatively straightforward. For Chinese, a word segmentation tool (such as Jieba or HanLP) is needed to split continuous Chinese text into individual words. For example, segmenting "Beijing Bank Financial Technology Co., Ltd." into ['Beijing Bank', 'Financial', 'Technology', 'Limited Liability', 'Company'].
[0038] Stop Word Removal: Remove words that appear frequently but carry little information (such as "的", "了", "和", "in", "the", "and") to reduce noise and highlight key information.
[0039] Stemming and Lemmatization: Reduce words to their root forms. For example: "running" -> "run", "better" -> "good". This helps to treat words in different forms as the same entity.
[0040] Step 2: Text Vectorization Goal: Convert human-readable text into a numerical form (i.e., vector / high-dimensional array) that machines can understand and compute, while preserving semantic information as much as possible. This is the core of how AI understands text.
[0041] Traditional methods generally use TF-IDF (Term Frequency - Inverse Document Frequency) to represent text as a vector based on word weights. It can measure the importance of a word in a document but cannot capture word order and deep semantics.
[0042] AI methods use Word2Vec / GloVe to map words to vectors, where semantically similar words are closer in the vector space. Or use BERT / Sentence-BERT (SBERT). These powerful pre-trained models can map an entire sentence or phrase to a fixed-length vector. Their core advantage is that for sentences with similar semantics, their vectors are closer in space (usually measured by cosine similarity). For example: The cosine similarity between the vector of ("苹果手机") and the vector of ("iPhone") will be very high. The similarity between the vector of ("苹果手机") and the vector of ("吃苹果") will be very low.
[0043] Step 3: Similarity Calculation and Candidate Pair Generation Goal: Efficiently calculate the similarity scores between text pairs and screen out the most likely matching candidate pairs to avoid the huge computational overhead of full-scale comparison.
[0044] Blocking: This is the key to dealing with large-scale data. Group the data by a certain coarse-grained key and only perform comparisons within the groups, greatly reducing the amount of computation.
[0045] Common blocking keys: initials of pinyin, postal codes, industry categories, keywords, etc. Example: When comparing company names, you can use "city name" as a blocking key, so that only companies within the same city will be compared.
[0046] Similarity calculation: For two texts in a candidate pair, calculate the similarity between their vector representations. Cosine similarity is used to measure the directional consistency between two vectors. The result is between [-1, 1], usually taking values in the range [0, 1]. The larger the value, the more similar the vectors are.
[0047] Step 4: Matching Decisions Objective: Based on the calculated similarity score, determine whether two texts refer to the same entity.
[0048] Traditional methods directly set a similarity threshold (e.g., 0.9). A score above the threshold is considered a "match," and below it is considered a "mismatch." This method is simple but inflexible. We use an AI machine learning classifier to prepare a labeled training set (explicitly knowing which text pairs match and which don't). We input the features of the text pairs (such as similarity scores across multiple fields, the vectors themselves, etc.) into the classification model for training. Common models include gradient boosting trees, logistic regression, support vector machines, or dedicated deep Siamese networks. The model outputs a matching probability (e.g., 0.95), and we make the final decision based on the set probability threshold (e.g., 0.9). This method can learn more complex non-linear relationships.
[0049] Step 5: Post-processing and evaluation Objective: To optimize the matching results and scientifically evaluate the performance of the entire system.
[0050] Conflict resolution: When merging two matching text records, how to handle conflicts in corresponding fields? Example: One record's address is "Beijing Haidian District", and the other is "Beijing Haidian District". When merging, the more standardized or complete format should be selected.
[0051] Evaluation indicators: Precision: Of the candidate pairs judged as "matches" by the system, how many are actually matches? (How accurately are they found?) Recall: Of all truly matching candidate pairs, the system successfully identifies the majority of them. (How many pairs were found?) F1-Score: The harmonic mean of precision and recall, which is a comprehensive metric. Feedback loop: Add cases where the system misjudged (especially those with high confidence but incorrect judgments) to the training set, retrain the model, and continuously iterate and optimize it.
[0052] Beneficial effects: Addressing the issue of missing semantic understanding: Traditional methods cannot identify synonymous fields such as "customer name" and "user name", while AI achieves intelligent field alignment through natural language processing and semantic embedding models (such as BERT), significantly improving matching accuracy.
[0053] Overcoming the problems of rule dependence and high manual costs: Without the need to pre-set complex matching rules, AI can automatically learn data patterns and mapping relationships, reducing manual configuration and adapting to dynamically changing data structures.
[0054] Improved tolerance to format differences and noise: AI can automatically identify and normalize different formats such as dates and amounts (e.g., "2023-01-01" vs. "01 / 01 / 2023"), and robustly handle noise such as spelling errors and abbreviations.
[0055] Supports comparison of unstructured and multimodal data: Traditional methods are limited to structured data, while AI can process data in text, logs, JSON, XML and even images to achieve cross-modal content comparison.
[0056] Reduce false alarms and false negatives: Through contextual understanding and difference importance assessment models, AI can distinguish between "critical changes" (such as monetary errors) and "irrelevant differences" (such as note updates), reducing invalid alarms.
[0057] Achieving automation and scalability: AI models can process massive amounts of heterogeneous data in batches, be integrated into CI / CD or data pipelines, and support real-time or periodic automatic comparisons to improve efficiency and coverage.
[0058] It has the ability to learn and continuously optimize itself: Through feedback mechanisms, AI can learn from human corrections, continuously optimize matching strategies, adapt to business evolution, and form a closed-loop optimization.
[0059] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An AI-based data comparison method, characterized in that, The data comparison method includes: Data preprocessing involves cleaning and standardizing the raw text into regular text. Text vectorization processing converts the regularized text into a numerical form that machines can understand and calculate; Similarity calculation and candidate pair generation; Matching decision: Based on the calculated similarity score, determine whether two texts refer to the same entity and obtain the matching result; Post-processing and evaluation involve optimizing the matching results and scientifically evaluating the performance of the entire system.
2. The AI-based data comparison method according to claim 1, characterized in that, The data preprocessing, which cleans and standardizes the original text into regularized text, specifically includes: Cleaning: Remove irrelevant characters: Remove HTML tags, special characters, extra spaces, and line breaks; Handling missing values: Filling in or deleting fields with missing text; Standardization: Case unification: Convert all characters to lowercase; Abbreviations and abbreviation extensions: restoring common abbreviations to their full forms; Standardized format: Standardize the format of dates, numbers, and units; Word segmentation: For English, word segmentation by spaces and punctuation is relatively simple; For Chinese, word segmentation tools are needed to break down continuous Chinese text into individual words; Remove stop words: Remove frequently occurring words that contain little information to highlight key information; Stem extraction and word form reduction: Reducing words to their root form.
3. The AI-based data comparison method according to claim 1, characterized in that, The text vectorization process, which converts the regularized text into a numerical form that machines can understand and calculate, specifically includes: Word2Vec / GloVe is used to map words to vectors, with semantically similar words being closer together in the vector space; an entire sentence or phrase is mapped to a fixed-length vector.
4. The AI-based data comparison method according to claim 1, characterized in that, The similarity calculation and candidate pair generation specifically include: calculating the similarity score between text pairs and selecting the candidate pairs that are most likely to match.
5. The AI-based data comparison method according to claim 4, characterized in that, The calculation of similarity scores between text pairs and the selection of the most likely matching candidate pairs specifically includes: Blocking: Grouping data using a coarse-grained key and comparing only within the group; Commonly used blocking keys: first letter of Pinyin, postal code, industry category, keywords; Similarity calculation: For two texts in a candidate pair, calculate the similarity between their vector representations; use cosine similarity to measure the directional consistency of the two vectors, the result is between [-1, 1], and the value is between [0, 1], the larger the value, the more similar they are.
6. The AI-based data comparison method according to claim 1, characterized in that, The matching decision, based on the calculated similarity score, determines whether two texts refer to the same entity, and the specific matching result includes: If the score is higher than the threshold, it is considered a match; if it is lower, it is considered a non-match. Prepare a labeled training set using an AI machine learning classifier; Input the features of the text pairs into the classification model for training; Commonly used models include: gradient boosting trees, logistic regression, support vector machines, or deep Siamese networks; The model outputs a matching probability and makes a final decision based on the set probability threshold.
7. The AI-based data comparison method according to claim 6, characterized in that, The features of the text pair specifically include: similarity scores of multiple fields and the vector itself.
8. The AI-based data comparison method according to claim 1, characterized in that, The post-processing and evaluation, which optimizes the matching results and scientifically evaluates the performance of the entire system, specifically includes: Evaluation indicators include: Precision: Of the candidate pairs that the system judges as "matches", how many are actually matches; Recall: Of all truly matching candidate pairs, the number successfully identified by the system. F1-Score: The harmonic mean of precision and recall, which is a comprehensive metric. Feedback loop: Add cases where the system misjudged to the training set, retrain the model, and continuously iterate and optimize.