An enterprise information intelligent cleaning method and system based on multi-source data fusion

Abstract

The application relates to the technical field of data processing, and discloses an enterprise information intelligent cleaning method and system based on multi-source data fusion, wherein the system comprises a data access layer, a data processing layer, a knowledge support layer and a service output layer. The data access layer accesses multi-source heterogeneous enterprise data through a configuration connector plug-in, and outputs standardized records after Schema mapping, format normalization and structured identification field checking; the data processing layer comprises an entity recognition and deduplication module, a multi-source data fusion and conflict resolution module and an intelligent cleaning and quality repair module, and sequentially performs cross-source entity alignment, field-level conflict resolution and multi-method quality repair; the knowledge support layer provides enterprise entity knowledge base, data source credibility score storage and other query services; the service output layer provides data query interfaces, quality report interfaces and manual review workstations; the system further comprises a feedback closed loop interface, and error records returned by downstream applications are used for model incremental training and data source credibility adjustment.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to an intelligent cleaning method and system for enterprise information based on multi-source data fusion. Background Technology

[0002] Enterprise-related data is widely distributed across various heterogeneous data sources, including business registration databases, tax information systems, judicial announcement platforms, publicly available internet data, and third-party data service providers. Existing technologies typically employ predefined field mapping rules and a single priority overlay strategy to aggregate these data sources, and then use rule engines or single models to correct data quality defects.

[0003] However, the aforementioned existing technologies have the following problems: First, different data sources have significant differences in field naming and format specifications for the same enterprise entity, making direct cross-source comparison and fusion impossible; Second, there is a lack of effective cross-source entity alignment methods, resulting in a large accumulation of duplicate records for the same enterprise entity after aggregation; Third, for the same field, the values ​​provided by different data sources conflict with each other, and the reliability of each data source varies, with existing technologies lacking a systematic conflict resolution approach; Fourth, the data contains various quality defects such as format errors, missing fields, outliers, and typos, which are difficult to fully cover with a single cleaning method; Fifth, the system lacks the ability to quantitatively assess the data quality status and cannot incorporate downstream business feedback into the closed-loop path of continuous system optimization, resulting in the cleaning capability becoming rigid and unable to continuously improve with business accumulation. Summary of the Invention

[0004] This invention provides an intelligent cleaning method and system for enterprise information based on multi-source data fusion, which solves the technical problems in related technologies such as entity duplication, field conflict, low data quality and lack of continuous optimization capabilities in multi-source heterogeneous enterprise data.

[0005] This invention discloses an intelligent enterprise information cleaning system based on multi-source data fusion, comprising: a data access layer configured to acquire multi-source heterogeneous enterprise raw data through a data source connector, and output standardized records conforming to a standard enterprise information data model after schema mapping and format normalization processing; an entity recognition and deduplication module configured to generate name semantic feature vectors, structural attribute feature vectors, and unique identifier precise matching features for standardized enterprise records, calculate comprehensive similarity, and perform hierarchical clustering to form enterprise entity clusters; a multi-source data fusion and conflict resolution module configured to perform field-level differential conflict resolution on records from different data sources within the same entity cluster based on dynamic credibility scoring of data sources; an intelligent cleaning and quality repair module configured to perform rule verification and machine learning model detection and correction on the fused enterprise records; and a feedback closed-loop interface configured to receive erroneous records returned by downstream application systems and trigger incremental model training and data source credibility adjustment.

[0006] Furthermore, the data access layer also includes a structured identifier field verification module. This module performs check digit verification on the unified social credit code according to the GB32100-2015 standard, and the check digit value... The calculation formula is: ; in The first 17 characters The position index of a bit character in the character set For the first The weighting factor corresponding to the position, when The checksum is taken from the character with index 0 in the character set; otherwise, it is taken from the character with index 1. The structured identifier field verification module filters out records where both the unified social credit code and organization code fail verification and the enterprise name field is empty, and classifies them into the noise record pool.

[0007] Furthermore, the calculation of the comprehensive similarity in the entity recognition and deduplication module includes: performing an exact identifier matching judgment; if there are identifier pairs with the same field name and equal field value in the exact identifier matching feature set of two records, then the comprehensive similarity is directly determined to be 1.0; if an exact match cannot be determined, cosine similarity is calculated based on the name semantic feature vector and the structural attribute feature vector respectively to obtain the name semantic similarity. Similarity with structural attributes The overall similarity is calculated using the following formula: ; in To assess overall similarity, and These are the weight coefficients for semantic similarity of names and structural attribute similarity, respectively, satisfying... The hierarchical clustering and merging process employs locality-sensitive hashing (LSH) to filter candidate alignment sets of enterprise entity feature vectors. Then, a hierarchical clustering algorithm based on a fully linked strategy is used to merge records with a comprehensive similarity exceeding an adaptive threshold into enterprise entity clusters.

[0008] Furthermore, the dynamic credibility score of the data source is determined by an authority level score. Historical accuracy score Data update frequency rating and field coverage completeness score The formula for calculating the overall credibility score, which is a weighted composite of four dimensions, is as follows: ; in To assess overall credibility, , , , These are the weight coefficients for the four dimensions, with the sum of each weight being 1; the data update frequency score. The calculation formula is: ; in It is a natural constant. This represents the number of days since the last data update of this data source. This is the attenuation coefficient.

[0009] Furthermore, the field-level differential conflict resolution includes: for the identifying field, calculating the overall support for each candidate value. ,in To provide candidate values The collection of data sources For data source The overall credibility score selects candidate values ​​with the highest overall support and from data sources with an authority level score no lower than the set value as the fusion result; for the timeliness field, the score is weighted by timeliness. The candidate value with the highest score is selected as the fusion result, where The difference between the data timestamp of the candidate value and the current time. The timeliness weighting coefficient is used; for numerical fields, an outlier detection method based on interquartile range is used to filter outlier candidate values, and then the fusion value is calculated by weighting the comprehensive credibility score.

[0010] Furthermore, the intelligent cleaning and quality repair module includes: a rule engine cleaning submodule, which loads a configurable rule base to perform format verification and repair actions on enterprise records; and a machine learning model cleaning submodule, which includes using a character-level sequence labeling model to perform typo detection and correction on enterprise names, using an address semantic understanding model to perform hierarchical standard parsing on unstructured address text, and using the isolated forest algorithm to perform outlier detection on numerical fields, with the isolated forest algorithm outputting anomaly scores. ,in The sample to be tested. This represents the total number of samples within the current group. Let be the expected value of the path length among multiple isolated trees. For a sample size of The average path length of the binary search tree; the missing value filling submodule performs missing field filling according to the priority order of multi-source information completion and statistical inference of similar enterprises.

[0011] Furthermore, it also includes a data quality assessment module, which quantitatively scores the cleaned dataset from five dimensions: completeness, accuracy, consistency, timeliness, and uniqueness. The comprehensive data quality score is calculated using the following formula:

[0012] in To provide a comprehensive data quality score, To score for completeness, For accuracy scoring, For consistency scoring, For timeliness scoring, For uniqueness scoring, , , , , These are the weights for each dimension, and the sum of the weights for each dimension is 1.

[0013] Furthermore, the data access layer also includes a time-series marker appending processing module, configured to append data collection timestamps and data source update timestamps to standardized enterprise records; it also includes a single-source time-series record chain generation module, configured to use the data source identifier and the enterprise unique identifier as grouping keys, arrange records from the same data source in different collection batches for the same grouping key in ascending order of data source update timestamps to form a single-source time-series record chain, and when inserting a new node, compare each field value with the previous node field by field, and add 1 to the corresponding component of the cumulative change count vector for fields that have changed.

[0014] Furthermore, the entity recognition and deduplication module introduces a time-series confidence decay factor into the structural attribute similarity when calculating the comprehensive similarity. Decay factor for each field dimension The calculation formula is: ; in It is a natural constant. The attenuation intensity coefficient, and These are two records in their respective single-source time-series record chains. The cumulative number of changes for each field dimension; the weighted structural attribute similarity after time-series confidence decay adjustment is: ; in Let be the dimension number of the structural attribute feature vector. and These are the first two L2-normalized structural attribute feature vectors of the two records. Each component; for candidate record pairs located in the cluster boundary region, perform time-series chain comparison operation, align the two single-source time-series record chains according to the time window, calculate the comprehensive similarity of each time interval to form a similarity time-series sequence, perform Spearman rank correlation coefficient test on the sequence, when the rank correlation coefficient is lower than the negative correlation threshold and the end comprehensive similarity is lower than the set lower bound, it is determined to be feature drift caused by attribute change and retained in the same entity cluster.

[0015] This invention discloses an intelligent enterprise information cleaning method based on multi-source data fusion, used to execute the aforementioned intelligent enterprise information cleaning system based on multi-source data fusion, comprising: Step 1, acquiring multi-source heterogeneous enterprise raw data through a data source connector, and outputting standardized records conforming to a standard enterprise information data model after schema mapping and format normalization processing; Step 2, generating name semantic feature vectors, structural attribute feature vectors, and unique identifier precise matching features for the standardized enterprise records, calculating comprehensive similarity, and performing hierarchical clustering to form enterprise entity clusters; Step 3, performing field-level differential conflict resolution on records from different data sources within the same entity cluster based on dynamic credibility scoring of the data source; Step 4, performing rule verification and machine learning model detection and correction on the fused enterprise records; Step 5, receiving erroneous records returned by downstream application systems and triggering incremental model training and data source credibility adjustment.

[0016] The present invention has the following beneficial effects: First, by configuring a unified data source connector framework in the data access layer, along with a schema mapping processing module and a format standardization processing module, raw data from multiple heterogeneous data sources can be uniformly converted into a standard enterprise information data model format. This eliminates the heterogeneity barriers of multi-source data in terms of field definition and format standardization, providing a unified data foundation for subsequent cross-source entity alignment and data fusion.

[0017] Second, by fusing three complementary features—name semantic feature vectors, structural attribute feature vectors, and unique identifier precise matching features—and employing locality-sensitive hashing (LSH) to filter candidate alignment sets before performing hierarchical clustering, this approach can effectively identify records of the same enterprise entity from different data sources on large-scale datasets, reducing computational complexity from... Reduced to approximately ,in The system records the total number of enterprises, and the three complementary features cover the main difficult scenarios in entity alignment, solving the problem of duplicate existence of the same enterprise entity in multi-source data.

[0018] Third, by introducing a dynamic credibility score based on a weighted composite of four dimensions—authority level, historical accuracy, data update frequency, and field coverage completeness—and by employing differentiated resolution strategies for identifiable fields, time-sensitive fields, numerical fields, and descriptive text fields, the system can make verifiable resolution decisions based on the actual credibility of the data source and the freshness of the data when multiple source field values ​​conflict. This solves the problem of lacking a systematic basis for resolution when multiple source data field conflicts occur.

[0019] Fourth, by combining rule engine cleaning with cleaning using multiple specialized machine learning models, the rule engine handles format errors and compliance defects with clear business definitions, the sequence labeling model handles typos in company names and OCR recognition errors, the address resolution model handles the standardized parsing of unstructured address text, and the isolated forest algorithm handles outlier detection for numerical fields. The data quality defect types covered by these various cleaning methods complement each other, overcoming the limitation that a single cleaning method cannot comprehensively cover multiple types of data quality defects.

[0020] Fifth, by using the feedback closed-loop interface, the error records returned by the downstream application system are used to trigger the incremental training and update of the machine learning model and to make real-time adjustments to the historical accuracy statistics of the relevant data sources. The accuracy of the system's cleaning model and the accuracy of the comprehensive credibility score of the data sources continue to improve as the system continues to run, solving the problem of the system's cleaning capabilities being fixed and unable to be continuously optimized with the accumulation of business.

[0021] Sixth, in Example 2, by introducing a time-series tagging additional processing module and a single-source time-series record chain generation module, and by applying a negative exponential decay factor to the field dimensions that have undergone multiple historical changes in entity similarity calculation, the system can adaptively perceive the temporal stability of each field dimension at the feature weight level, so that the feature differences caused by normal changes in enterprise attributes will not be mistakenly amplified as the basis for judging different entities; furthermore, by performing a time-series chain comparison operation based on Spearman rank correlation coefficient on candidate record pairs in the cluster boundary region, the system can distinguish between the same entity whose current comprehensive similarity has decreased due to gradual attribute changes and different entities that happen to have similar attributes, thus solving the problem that fixed-weight comprehensive similarity calculation cannot cope with the temporal evolution of enterprise attributes. Attached Figure Description

[0022] Figure 1 This is a flowchart of the entity similarity calculation method provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the overall architecture of the multi-source heterogeneous data cleaning pipeline provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the entity recognition and deduplication module processing flowchart provided in an embodiment of the present invention; Figure 4 This is a schematic diagram comparing the four dimensions of the comprehensive credibility score of various data sources provided in the embodiments of the present invention; Figure 5 This is a schematic diagram showing the summary and comparison of the comprehensive credibility scores of various data sources provided in the embodiments of the present invention; Figure 6 This is a schematic diagram of the outlier filtering and weighted fusion process of the registered capital field provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the five-dimensional scoring and weighted contribution of data quality in this batch provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the decision-making process for field-level conflict resolution strategies provided in an embodiment of the present invention; Figure 9 This is a schematic diagram illustrating the dynamic update of the data source credibility score by the feedback closed loop provided in this embodiment of the invention. Detailed Implementation Example

[0023] like Figures 1-9 As shown in the embodiment, an enterprise information intelligent cleaning system based on multi-source data fusion is provided. The enterprise information intelligent cleaning system based on multi-source data fusion is divided into four layers in terms of architecture: data access layer, data processing layer, knowledge support layer, and service output layer. Each layer communicates with each other through standardized internal interfaces to jointly support the complete operation of the enterprise information intelligent cleaning pipeline.

[0024] The data access layer is responsible for the unified access, format conversion, and preliminary filtering of multi-source heterogeneous data; the data processing layer carries the core processing logic such as entity recognition and deduplication, data fusion and resolution, and intelligent cleaning and repair; the knowledge support layer operates as an independent service, including an enterprise entity knowledge base, data source credibility score storage, industry classification system retrieval service, and geocoding dictionary, providing query services for each module of the data processing layer; the service output layer provides standardized data query interfaces, data quality report interfaces, and a manual review workbench, supporting the data service needs of various downstream application scenarios.

[0025] The data access layer is configured to acquire raw data from multiple heterogeneous enterprises and output standardized records that conform to the standard enterprise information data model format. The data access layer is deployed at the front end of the system. It contains a unified data access adaptation framework, which adopts a configurable data source connector structure, encapsulating the access logic of each type of data source into an independent connector plugin. Each connector plugin implements a uniformly defined connector interface specification, which includes four standard methods: connection initialization, data retrieval, incremental synchronization, and connection closure.

[0026] For the business registration database, the connector plugin retrieves data via direct database connection or official open interfaces, supporting both full synchronization and incremental synchronization based on change timestamps. For the tax information system, the connector plugin acquires data in batches periodically through authorized data sharing interfaces, with interface call frequency and data range configured according to the data sharing protocol. For the judicial announcement platform, the connector plugin collects information such as court announcements, lists of dishonest judgment debtors, and judicial auctions through a combination of structured webpage parsing and interface integration, with collection tasks running on a scheduled basis. For publicly available internet data, the data access layer integrates a distributed web crawler component, which has built-in anti-crawling strategies and collection frequency control structures, supporting the collection of structured data from publicly available sources such as official enterprise websites and mainstream enterprise information platforms. For third-party data service provider interfaces, the connector plugin supports two mainstream protocols: RESTful and SOAP interfaces, enabling rapid access by configuring the service provider interface address, authentication parameters, and data parsing rules. All connector plugins support parameterized management via configuration files, with access configuration information stored in the system configuration center and supporting dynamic hot reloading.

[0027] The schema mapping processing module is configured to align the original fields of each data source to the standard enterprise information data model. The system predefines a standard enterprise information data model, encompassing a set of fields across five dimensions: basic enterprise identification information, business registration information, financial information, legal information, and relationship information. The schema mapping processing module loads a schema mapping configuration file stored in JSON format at runtime. This configuration file describes the correspondence between the original fields of the data source and the standard fields of the standard enterprise information data model, supporting three mapping types: one-to-one mapping, many-to-one merge mapping, and one-to-many split mapping. It also supports embedding data transformation expressions within the mapping rules. The schema mapping processing module extracts and transforms fields for each original data record according to the mapping rules, outputting standardized records conforming to the standard enterprise information data model format. For extended fields present in the original data source but not defined in the standard enterprise information data model, they are stored as reserved extended attribute fields in the form of JSON objects, ensuring no loss of original information.

[0028] The format normalization module is configured to perform uniform normalization processing on the values ​​of characters, numbers, and dates in each field of a standardized record; The format normalization module performs three types of normalization processing on the field values ​​in standardized records. Character normalization includes converting full-width characters to half-width characters, pruning leading and trailing whitespace, compressing consecutive whitespace characters, and filtering and removing special control characters, while uniformly converting all text field encoding formats to UTF-8. Numeric field normalization removes currency symbols, thousands separators, and extra spaces from numeric fields such as registered capital, and converts Chinese numerals to Arabic numerals. Date field normalization converts various common date formats to the ISO8601 standard format, marking unparseable date strings as format anomalies.

[0029] The structured identifier field validation module is configured to perform format compliance validation and filter noisy records for the Unified Social Credit Code and Organization Code; The structured identifier field verification module performs format compliance verification on structured identifier fields such as the Unified Social Credit Code and Organization Code.

[0030] For the Unified Social Credit Code, the structured identifier field verification module performs 18-bit code check digit verification according to the GB32100-2015 standard. The calculation method for the check digit value is as follows: Let the numerical sequence corresponding to the first 17 characters be... ,in The first 17 characters The position index of a bit character in the character set The range of values ​​is to The character set consists of 31 characters: "0123456789ABCDEFGHJKLMNPQRTUWXY". Each character's value represents its index within the character set (counting from 0). The weighted factor sequence... The sequence is 3, 7, 9, 10, 5, 8, 4, 2, 1, 6, 3, 7, 9, 10, 5, 8, 4, where... For the first The weighting factor corresponding to the bit, then the check code value The calculation formula is: ; in The first 17 characters The position index of a bit character in the character set For the first The weighting factor corresponding to each position. To find the upper bound of the sum, which is the total number of the first 17 characters. When The checksum is taken from the character with index 0 in the character set; otherwise, it is taken from the character with index 1. The structured identifier field verification module compares the calculated check code with the 18th digit of the code. If they do not match, the unified social credit code field of the record is marked as failing format verification.

[0031] It should be noted that, The range of values ​​is ,therefore The range of values ​​is When the modulo operation result is 0, At this point, the checksum is taken from the character with index 0 in the character set; when the modulo operation result is not 0, The checksum is taken from the character set index. The index range for the specified characters is always within the valid range of the character set. Inside.

[0032] For organization codes, the structured identifier field verification module performs 9-bit code verification according to the GB / T11714-1997 standard.

[0033] The structured identifier field validation module filters out records that fail to validate both the unified social credit code and the organization code and whose enterprise name field is empty. These records are then placed in the noise record pool and stored separately. The remaining records are processed by the data processing layer.

[0034] The entity recognition and deduplication module is configured to perform cross-source entity alignment on standardized enterprise records and merge records of the same enterprise entity into entity clusters; The data processing layer is the core computing layer of the system, which includes an entity recognition and deduplication module, a multi-source data fusion and conflict resolution module, and an intelligent cleaning and quality repair module. Each module runs in series in a pipeline manner, and also supports parallel batch processing mode for specific data sources.

[0035] The Enterprise Entity Feature Vector Generation Submodule is configured to generate a name semantic feature vector, structural attribute feature vector, and a set of precise matching features for unique identifiers for each standardized enterprise record. The Enterprise Entity Feature Vector Generation Submodule generates three types of features for each standardized enterprise record.

[0036] The first category is the semantic feature vector of the enterprise name. The enterprise entity feature vector generation submodule uses a pre-trained Chinese language model (Chinese version of BERT or RoBERTa) to semantically encode the enterprise name field. Its input is the character sequence of the enterprise name, and its output is the vector corresponding to the classification label in the last hidden state of the model. The vector is then subjected to L2 normalization and used as the semantic feature vector of the enterprise name.

[0037] The second category is structural attribute feature vectors. The enterprise entity feature vector generation submodule takes four structured attribute fields as input: the administrative division code of the registered location, the range of registered capital, the industry classification code, and the year of establishment. The administrative division code of the registered location, the range of registered capital, and the industry classification code are mapped into binary vector components using one-hot encoding. The year of establishment is processed into numerical components using decimal scaling normalization. The components are concatenated to form a complete structural attribute feature vector, and L2 normalization is also performed on it.

[0038] The third category is precise matching features based on unique identifiers. The enterprise entity feature vector generation submodule extracts the values ​​of three types of unique identifier fields: unified social credit code, organization code, and business registration number, forming a set of precise matching features.

[0039] The entity similarity calculation submodule is configured to calculate the overall similarity for any two enterprise records. The entity similarity calculation submodule calculates the overall similarity for any two enterprise records in three steps.

[0040] Step 1: Perform exact identifier matching judgment, checking whether there are identifier pairs with the same field name and equal field value between the exact identifier matching feature sets of the two records. If at least one valid exact identifier match exists, the overall similarity is directly determined to be 1.0.

[0041] Step two: When an exact match cannot be determined, calculate both name semantic similarity and structural attribute similarity using cosine similarity. The inputs are the name semantic feature vector and the structural attribute feature vector, respectively, and the outputs are the name semantic similarity and the structural attribute feature vector, respectively. Similarity with structural attributes .

[0042] Step 3: Weight and fuse the semantic similarity and structural attribute similarity of the names. The formula for calculating the comprehensive similarity is as follows:

[0043] in To assess overall similarity, For name semantic similarity, For structural attribute similarity, and These are the weight coefficients for semantic similarity of names and structural attribute similarity, respectively, satisfying... System default configuration , The aforementioned weight parameters can be adjusted via the system configuration file.

[0044] The hierarchical clustering and merging submodule is configured to perform candidate set filtering and hierarchical clustering and merging on enterprise records to form enterprise entity clusters; Before performing full similarity calculation, the hierarchical clustering and merging submodule first uses locality-sensitive hashing (LSH) to quickly filter candidate alignment sets for enterprise entity feature vectors. It only calculates the exact similarity for records co-occurring within the hash bucket, thus reducing computational complexity from... Reduced to approximately ,in Record the total number of companies.

[0045] For the selected candidate record pairs, the hierarchical clustering and merging submodule uses a hierarchical clustering algorithm based on a fully linked strategy to cluster the enterprise records. The input is the comprehensive similarity of the candidate record pairs, and the output is the enterprise entity cluster formed by merging records with comprehensive similarity exceeding an adaptive threshold. The adaptive threshold is determined as follows: in the initial stage, the system trains a threshold selector based on a set of manually labeled enterprise entity aligned samples, aiming to determine the initial threshold with the optimal F1 score; during system operation, as the review results in the manual review queue accumulate, the system retrains the threshold selector with newly labeled data every preset period, dynamically updating the threshold.

[0046] The human-machine collaboration confirmation submodule is configured to push low-confidence candidate alignment results to human review and incorporate the review results into incremental learning. For candidate record pairs whose overall similarity falls within the low confidence interval, the human-machine collaborative confirmation submodule marks them as low-confidence candidate alignment results and pushes them to the review queue of the manual review workbench. The upper bound of the low confidence interval is an adaptive threshold, and the lower bound is the adaptive threshold minus the preset confidence interval width. The confidence interval width parameter can be adjusted through the system configuration file. Candidate record pairs whose overall similarity is lower than the lower bound of the low confidence interval are directly determined as different entities and do not enter the manual review queue. The manual review workbench displays detailed information and similarity calculation basis for the two candidate aligned records in a visual interface. Reviewers confirm or reject the alignment result through interface operations. The system processes the review results in real time: record pairs confirmed as the same entity are added to the positive sample training pool, and rejected record pairs are added to the negative sample training pool. After the training pool reaches the preset batch size, online incremental learning of the model is triggered to update the fine-tuning parameters of the semantic encoding model and the threshold selector parameters.

[0047] The multi-source data fusion and conflict resolution module is configured to perform field-level fusion and conflict resolution on records from different data sources within the same enterprise entity cluster. The data source dynamic credibility scoring submodule is configured to calculate and maintain a comprehensive credibility score for each data source; The data source dynamic credibility scoring submodule maintains a dynamic credibility score for each data source. The dynamic credibility score is a weighted composite of four dimensions: authority level score, historical accuracy score, data update frequency score, and field coverage completeness score.

[0048] The authority rating is based on a fixed score set according to the data source type: 1.0 for official government data sources, 0.8 for authoritative industry databases, 0.6 for commercial data service providers, and 0.4 for publicly available internet data.

[0049] Historical accuracy score is calculated based on the statistical results of the verification of the data source field value during the system's historical operation. The calculation formula is as follows: ; in Rate historical accuracy. This represents the total number of times the field value of this data source has been confirmed as correct in historical verifications. This represents the total number of times the field values ​​from this data source were verified. (This is during initial system startup.) Use the default value of 0.8.

[0050] The data update frequency score is calculated based on the data update cycle of the data source, using the following formula: ; in Rate the frequency of data updates. It is a natural constant. This represents the number of days since the last data update of this data source. This is the attenuation coefficient, with a default value of 0.01.

[0051] The field coverage completeness score is calculated by determining the proportion of non-empty fields covered by the data source in all fields of the standard enterprise information data model. The formula is as follows: ; in Score the field coverage completeness. The number of non-null fields provided for this data source. This represents the total number of fields in the standard enterprise information data model.

[0052] The formula for calculating the overall credibility score is: ; in To assess overall credibility, As an authoritative rating, Rate historical accuracy. Rate the frequency of data updates. Score the field coverage completeness. , , , These are the weight coefficients for the four dimensions, with the sum of each weight being 1. The scores for all four dimensions are normalized to [value missing]. Dimensionless values ​​within an interval can be directly summed using weighted calculations. The dynamic credibility scoring submodule for data sources recalculates the overall credibility score for all data sources every 24 hours.

[0053] The field-level conflict resolution submodule is configured to use differentiated strategies to perform conflict resolution for different field types and record data lineage information; The field-level conflict resolution submodule employs differentiated resolution strategies for each field type.

[0054] For identifying fields (company name, unified social credit code, etc.), the field-level conflict resolution submodule counts the frequency of each candidate value in the data sources participating in the fusion and the comprehensive credibility score of the corresponding data sources. It calculates the comprehensive support of each candidate value and selects the candidate value with the highest comprehensive support and from at least one data source with an authority level score of not less than 0.8 as the fusion result; otherwise, the field is marked as pending verification. Among these, candidate values... Overall support The calculation formula is: ; in Candidate values Overall support To provide candidate values The collection of data sources for Data source in For data source The overall credibility score.

[0055] For fields with high time sensitivity (registered address, legal representative, etc.), the field-level conflict resolution submodule comprehensively considers the data source's overall credibility score and the data timestamp, and calculates the time-sensitive weighted score for each candidate value according to the formula: ; in Weighted scoring based on timeliness Assign a comprehensive credibility score to the data source to which the candidate value belongs. The difference (in days) between the data timestamp of the candidate value and the current time. This is the timeliness weighting coefficient, with a default value of 0.5. To normalize to Dimensionless score for the interval As a time-sensitive factor, when Its value range is ,and Since the dimensions are consistent, the two items can be directly weighted and summed. The candidate value with the highest timeliness-weighted score is selected as the fusion result.

[0056] For numeric fields (registered capital, number of employees, etc.), the field-level conflict resolution submodule first uses an outlier detection method based on interquartile range to filter obviously abnormal candidate values. The outlier judgment range is less than... or greater than ,in It is the first quartile. It is the third quartile. The interquartile range is used. For the remaining valid candidate values, a fusion strategy based on a comprehensive credibility score weighting is applied to calculate the fusion result, and the fusion value is... The calculation formula is: ; in For fusion value, This refers to the set of data sources to which the valid candidate values ​​belong after filtering for outliers. for Data source in For data source Provided candidate values, For data source The overall credibility score.

[0057] For descriptive text fields (business scope, business status, etc.), the field-level conflict resolution submodule segments the descriptive text provided by each data source into sentences or semantic fragments. It then uses a pre-trained language model to calculate the cosine similarity between these semantic fragments. Fragments with a cosine similarity exceeding a set threshold are considered semantically equivalent, and only the version from the data source with the highest overall credibility score is retained. Semantically inequivalent fragments are then concatenated and integrated after being sorted in descending order of the overall credibility score from the data source. The default cosine similarity threshold used to determine semantic equivalence is 0.85, which can be adjusted via a system configuration file.

[0058] The multi-source data fusion and conflict resolution module records complete data lineage information for each field value of each fusion record. The lineage record is stored in JSON format and includes a list of data sources participating in the fusion of that field, the type of conflict resolution strategy adopted, the final fusion result, and a summary of the decision basis.

[0059] The intelligent cleaning and quality repair module is configured to perform rule verification, model detection, and missing value filling on the merged enterprise records. The rules engine cleaning submodule is configured to load a configurable rule base and perform format validation and repair actions on enterprise records. The rule engine cleaning submodule has a built-in configurable rule engine. The rule base is stored and managed in XML format. Each rule includes attributes such as rule number, rule name, applicable fields, validation condition expression, remediation action, and severity level. Built-in rule categories include unified social credit code verification rules, administrative division code compliance check rules, registered capital format standardization rules, and date field legality verification rules. The rule base can be maintained through a visual rule management interface, and rule changes take effect immediately.

[0060] The machine learning model cleaning submodule is configured to perform detection and correction for typos in company names, address text, and numeric fields, respectively. The machine learning model cleaning submodule deploys three types of specialized models.

[0061] The enterprise name misspelling and OCR error detection and correction model adopts a character-level sequence labeling model. Its input layer receives the character sequence of the enterprise name, the hidden layer uses a bidirectional long short-term memory network to perform context encoding on the character sequence, and the output layer is a fully connected layer that outputs the labeling result (normal or needs correction) for each character position, along with the confidence distribution of candidate characters for correction. The model uses character-level cross-entropy loss as the loss function and is trained using the Adam optimization algorithm. The training data consists of manually labeled enterprise name sample pairs containing misspellings and OCR errors. Correction suggestions with a confidence level higher than 0.90 are automatically implemented; suggestions with a confidence level between 0.70 and 0.90 are pushed to a manual review queue; and suggestions with a confidence level lower than 0.70 are only marked as potentially erroneous.

[0062] The address information semantic understanding and standardized parsing model standardizes and parses unstructured address text in the registered address field, decomposing it into hierarchical structured address components such as province, city, district / county, street, and house number. It then matches each level of address element with the geocoding dictionary of the knowledge support layer, outputting a standardized structured address representation and the corresponding precise administrative division code.

[0063] The outlier detection algorithm employs the Isolation Forest algorithm, grouping data by industry classification code and administrative division code of registered location, to detect outliers in numeric fields within each group. The input to the Isolation Forest algorithm is the set of numeric field values ​​for each enterprise's records after grouping by industry and region; the output is an outlier score for each field value. The calculation formula is: ; in For the sample With a sample size of Abnormal scores at that time These are samples of numeric field values ​​to be tested. This represents the total number of samples within the current group. For the sample Path length in the isolation tree Let be the expected value of the path length among multiple isolated trees. For a sample size of The average path length of a binary search tree. The calculation formula is: ; in , for The function's argument, which takes the value here. , It is the natural logarithm function. This is Euler's constant (approximately 0.5772). Field values ​​with anomaly scores exceeding 0.75 are marked as suspected anomalies and pushed to the manual review queue.

[0064] The missing value filling submodule is configured to perform multi-source completion or statistical inference filling on missing fields according to priority order. The missing value filling submodule counts the missing values ​​in each field of the standard enterprise information data model and executes the filling strategy in order of priority.

[0065] The first priority is multi-source information completion: retrieve records of the same enterprise entity from other data sources in the enterprise entity knowledge base. If there is a valid value for the field in a data source with a comprehensive credibility score of not less than 0.7, then use that value directly for completion.

[0066] The second priority is statistical inference from similar enterprises: when no information is available from multiple sources, suggested completion values ​​are generated based on the historical statistical distribution characteristics of enterprises in the same industry and region. For numerical fields, the median of that field among similar enterprises is used; for categorical fields, the most frequent value is used. All suggested completion values ​​generated through statistical inference are marked with a confidence level and clearly marked as inferred completion values ​​in the data quality report.

[0067] The data quality assessment module is configured to calculate enterprise information data quality scores from five dimensions and generate quality reports. The data quality assessment module quantifies and scores the cleaned dataset across five dimensions: completeness, accuracy, consistency, timeliness, and uniqueness. All scores are normalized to [value missing]. The dimensionless ratio values ​​of the interval can be directly summed using weighted summation.

[0068] The formula for calculating the integrity score is: ; in To score for completeness, This is the collection of all enterprise records in the cleaned dataset. For set The number of Chinese companies recorded For a predefined set of key fields, For set The number of key fields in the middle. For set Enterprise records in For set Key fields in For indicator functions, record fields It takes the value 1 when it is not empty and 0 when it is empty.

[0069] The accuracy score is calculated as the percentage of field instances that pass all quality rule validations and are not detected as anomalies. The formula is as follows: ; in For accuracy scoring, This represents the number of field instances that have passed all quality rule validations and have not been flagged as an anomaly. This represents the total number of all field instances.

[0070] The consistency score is calculated by determining the percentage of enterprise records that satisfy all cross-field consistency constraints. The formula is as follows: ; in For consistency scoring, To satisfy the enterprise record set with all cross-field consistency constraints, For set The number of Chinese companies recorded This represents the total number of enterprise records in the cleaned dataset.

[0071] The timeliness score is calculated based on the difference between the data timestamp of each field and the current time, using the following formula: ; in For timeliness scoring, This represents the number of all enterprise records in the cleaned dataset. For the entire field set, For set The number of fields in the middle, For set Enterprise records in For set The fields in For record Middle field The difference between the data timestamp and the current time (in days). This is the time-related decay coefficient. It is a natural constant. The dimensionless decay factor is obtained by applying a negative exponential transformation to the time difference, when The range of values ​​is The scoring dimensions are consistent with those of other dimensions.

[0072] The formula for calculating the uniqueness score is: ; in For uniqueness scoring, This represents the number of enterprise entity clusters after entity deduplication. This represents the total number of enterprise records in the cleaned dataset.

[0073] The data quality assessment module synthesizes a comprehensive data quality score using the following formula:

[0074] in To provide a comprehensive data quality score, To score for completeness, For accuracy scoring, For consistency scoring, For timeliness scoring, For uniqueness scoring, , , , , These represent the weights for each dimension, and the sum of the weights for each dimension is 1. The data quality assessment module automatically generates a data quality report after each complete cleaning pipeline run, and the data quality report is output in both HTML and PDF formats.

[0075] The feedback loop interface is configured to receive error records returned by downstream application systems and trigger incremental model training and data source credibility adjustment. The feedback loop interface receives error records from downstream application systems. The received feedback data includes the enterprise identifier, error field name, error field value, and correct field value. Upon receiving the data, the system adds the field value information from the feedback record to the training data pool of the corresponding machine learning model, triggering incremental training updates. Simultaneously, it performs real-time adjustments (i.e., updates) to the historical accuracy statistics of the data source providing the error field values. middle and (Count the data), and write the feedback records into the data quality issue log.

[0076] In some embodiments, the pre-trained Chinese language model used by the enterprise entity feature vector generation submodule may be a domain-specific model fine-tuned with enterprise information domain corpus to enhance the semantic encoding capability for enterprise name domain-specific expressions.

[0077] In some embodiments, the weights of each dimension of the comprehensive credibility score in the data source dynamic credibility scoring submodule can be adjusted through the system configuration file, and the configuration can be optimized according to the distinguishing ability of each data source attribute under different business scenarios.

[0078] According to the embodiments of this implementation, by configuring a unified data source connector framework in the data access layer, and with the matching schema mapping processing module and format standardization processing module, the system can uniformly convert the raw data from heterogeneous data sources such as business registration databases, tax information systems, judicial announcement platforms, publicly available Internet data, and third-party data service providers into a standard enterprise information data model format. This eliminates the heterogeneity barriers of multi-source data in terms of field definition and format standardization, and provides a unified data foundation for subsequent cross-source entity alignment and data fusion.

[0079] By fusing three complementary feature classes—name semantic feature vector, structural attribute feature vector, and unique identifier precise matching feature—in the entity recognition and deduplication module, and then performing hierarchical clustering after filtering the candidate alignment set using locality-sensitive hashing, the system can effectively identify the same enterprise entity records from different data sources on large-scale datasets, reducing computational complexity from... Reduced to approximately ,in The system records the total number of enterprise records. Semantic encoding can capture situations where there are significant differences in word meaning but semantic equivalence, such as enterprise abbreviations, former names, and variations of common industry expressions. Structural attribute features and precise identifier matching provide independent verification paths from the attribute dimension and the identifier dimension. The three types of features complement each other and cover the main difficult scenarios in entity alignment, thereby solving the problem of the same enterprise entity existing repeatedly in different record forms in multi-source data.

[0080] By introducing a dynamic credibility score based on four dimensions—authority level, historical accuracy, data update frequency, and field coverage completeness—into the multi-source data fusion and conflict resolution module, and employing differentiated resolution strategies for identifiable fields, time-sensitive fields, numerical fields, and descriptive text fields, the system can make verifiable resolution decisions based on the actual credibility and freshness of the data source when multi-source field values ​​conflict, rather than relying on simple priority coverage or majority voting strategies. This solves the problem of lacking a systematic basis for resolving multi-source data field conflicts.

[0081] By combining rule engine cleaning with cleaning using multiple specialized machine learning models, the rule engine handles format errors and compliance defects with clear business definitions, the sequence labeling model handles typos in company names and OCR recognition errors, the address resolution model handles the standardized parsing of unstructured address text, and the isolated forest algorithm handles outlier detection for numerical fields. The data quality defect types covered by these cleaning methods complement each other, thus overcoming the limitation that a single cleaning method cannot comprehensively cover multiple types of data quality defects.

[0082] By using a feedback loop interface, the system sends back error records found by downstream application systems during business verification, triggering incremental training and updates of the machine learning model and making real-time adjustments to the historical accuracy statistics of relevant data sources. The system's cleaning model accuracy and the accuracy of the data source's overall credibility score continuously improve as the system continues to run, thus solving the problem of the system's cleaning capabilities becoming fixed and unable to be continuously optimized with business accumulation.

[0083] Example 2 Based on the enterprise information intelligent cleaning system based on multi-source data fusion provided in Example 1, the entity recognition and deduplication module uses a static weighted comprehensive similarity calculation method to match and determine candidate record pairs. After being accessed, the records from each data source are processed by the schema mapping processing module and the format standardization processing module and then uniformly enter the similarity calculation and clustering process without carrying temporal evolution information.

[0084] When there is a time lag in data updates from multiple data sources, the attribute values ​​of the same enterprise entity may undergo substantial changes (such as changes in registered capital, address relocation, or name change) within different time windows. This causes the comprehensive similarity calculation formula based on fixed feature weights to fail to distinguish between "the same entity after attribute changes" and "different entities with similar attributes." Consequently, the entity alignment results may incorrectly split the changed enterprise into multiple entities or incorrectly merge different enterprises due to historical attribute similarities. The adaptive threshold adjustment in Example 1 only optimizes the boundary of the comprehensive similarity value and fails to perceive the temporal evolution characteristics of attributes at the feature level, thus failing to distinguish between feature drift and entity differences.

[0085] A cross-source enterprise entity incremental alignment system based on hierarchical confidence decay and temporal chain comparison; According to an embodiment of this implementation, a cross-source enterprise entity incremental alignment method based on hierarchical confidence decay and temporal chain comparison is provided. Based on the system architecture of Embodiment 1, the cross-source enterprise entity incremental alignment method based on hierarchical confidence decay and temporal chain comparison provides targeted enhancements to the entity identification and deduplication modules in the data access layer and data processing layer. The structure and operation of the remaining data processing layer modules, knowledge support layer, and service output layer are the same as in Embodiment 1.

[0086] The time-series stamp appending processing module is configured to append data collection timestamps and data source update timestamps to standardized enterprise records. Building upon the connector plugin, schema mapping processing module, format normalization processing module, and structured identifier field validation module in Implementation Example 1, the data access layer further adds a time-series marker appending processing module. After the format normalization processing module completes the field value normalization output, the time-series marker appending processing module appends two time-series marker fields to each standardized enterprise record: a data acquisition timestamp field recording the system time when the connector plugin completed the data retrieval operation from the data source, with millisecond-level UTC timestamp precision; and a data source update timestamp field recording the time when the record was last modified by the data source in the original data source. If the original data source does not provide an update time, the data acquisition timestamp is used instead, and a data source update time missing flag is set for this field.

[0087] The single-source time-series record chain generation module is configured to organize multiple batches of records from the same data source into an ordered time-series chain based on the data source identifier and the enterprise's unique identifier. The single-source time-series record chain generation module uses the data source identifier and the enterprise's unique identifier (preferably the unified social credit code, then the organization code, and if both are missing, the business registration number) as grouping keys for records from the same data source in different collection batches. Records with the same grouping key from the same data source are then sorted in ascending order by the data source update timestamp, forming a single-source time-series record chain. The chain identifier of the single-source time-series record chain is a string combining the enterprise's unique identifier and the data source identifier, stored in the enterprise entity knowledge base of the knowledge support layer. Each record is stored as a node in the single-source time-series record chain, and nodes are linked in an ordered manner based on their adjacency according to timestamps.

[0088] When a new record is inserted as a new node into the corresponding single-source time-series record chain, the single-source time-series record chain generation module compares the field values ​​of the new node with the corresponding field values ​​of the previous node in the single-source time-series record chain field by field. For fields that have changed, 1 is added to the corresponding component of the cumulative change count vector, and the updated cumulative change count vector is persisted to the enterprise entity knowledge base along with the new node.

[0089] The enhanced enterprise entity feature vector generation submodule is configured to simultaneously extract change marker vectors and cumulative change count vectors based on the three types of features in Example 1; When generating the name semantic feature vector and structural attribute feature vector as described in Example 1, the enhanced enterprise entity feature vector generation submodule simultaneously reads the single-source time-series record chain to which the record belongs from the enterprise entity knowledge base and extracts the change marker vector and the cumulative change count vector.

[0090] The first change of the marker vector The i-th component corresponds to the i-th structural attribute feature vector. There are 10 field dimensions, among which For field dimension indexing: a value of 1 indicates that the field value has changed relative to the previous record in the same source time series chain, and a value of 0 indicates that no change has occurred. For the first record in a single source time series record chain, all components of the change marker vector are set to 0.

[0091] The first of the cumulative change count vectors Each component records the total number of changes to this field in the entire single-source time-series record chain, from the first record to the current record. This is a field-dimensional index, calculated by summing the components corresponding to the change marker vectors of each node in the single-source time-series record chain node by node. For records in a single-source time-series record chain containing only a single node, all components are set to 0, indicating that there is currently no observable change history.

[0092] An enhanced entity similarity calculation submodule is configured to introduce a time-series confidence decay factor into structural attribute similarity in the comprehensive similarity calculation. The enhanced entity similarity calculation submodule calculates the similarity between two cross-source records. and When calculating the overall similarity, based on the three-step calculation framework of Example 1, a time-series confidence decay factor is introduced into the structural attribute similarity calculation in the third step.

[0093] For the structural attribute feature vector of the th Each field dimension, decay factor The calculation formula is: ; in For the first Decay factor for each field dimension, It is a natural constant. The attenuation intensity coefficient can be adjusted via the system configuration file. and Records respectively and In their respective single-source time-series record chains, the first The cumulative number of changes for each field dimension. Index for field dimensions.

[0094] After time-series confidence decay adjustment, the effective weights of each field dimension are the original uniformly distributed weights multiplied by the corresponding decay factor. The normalized effective weights of each dimension after decay are then used to calculate the weighted structural attribute similarity. The calculation formula is as follows: ; in The weighted structural attribute similarity is adjusted for time-series confidence decay. Let be the dimension number of the structural attribute feature vector. For field dimension indexes and the value range is to , For the first Decay factor for each field dimension, and These are the first two L2-normalized structural attribute feature vectors of the two records. Each component.

[0095] The final formula for calculating the overall similarity has been adjusted to: ; in The overall similarity is adjusted for time-series confidence decay. For name semantic similarity, The weighted structural attribute similarity is adjusted for time-series confidence decay. , The definition and default value are the same as in Example 1. The logic for precise identifier matching in the first step is also the same as in Example 1. The enhanced entity similarity calculation submodule outputs a comprehensive similarity matrix adjusted for time-series confidence decay.

[0096] The enhanced hierarchical clustering and merging submodule is configured to perform temporal chain alignment operations on candidate record pairs in cluster boundary regions; The enhanced hierarchical clustering and merging submodule, after receiving the comprehensive similarity matrix and completing the local sensitive hash filtering and fully linked hierarchical clustering described in Example 1, additionally performs a temporal chain comparison operation on candidate record pairs located in the cluster boundary region. The cluster boundary region is defined as candidate record pairs whose comprehensive similarity falls between the adaptive threshold minus 0.15 and the adaptive threshold.

[0097] The specific steps of the time-series chain alignment operation are as follows: Step 1: Retrieve two records from the enterprise entity knowledge base of the knowledge support layer. and Each has its own single-source timing record chain.

[0098] Step two: Align the two single-source time-series record chains by timestamp, and divide each chain into several time intervals with a fixed time window length. For each time interval, take the feature vector of the latest record in that interval as the representative vector of that interval. If a single-source time-series record chain has no record in a certain time interval, fill it with the representative vector of the nearest preceding interval in that chain.

[0099] Step 3: Calculate the semantic similarity of the names and the weighted structural attribute similarity of the representative vectors of the two records for each time interval. and according to Example 1 , The weighted average similarity for this time interval is used to form a similarity time series. ,in They are respectively the 1st, 2nd to the 3rd The overall similarity of the effective time intervals, This represents the number of valid time intervals.

[0100] Step four involves performing a monotonically decreasing trend test on the similarity time series sequences, and using the Spearman rank correlation coefficient to measure the negative correlation between the similarity time series sequences and the time index sequences. The input to the Spearman rank correlation coefficient is the pairing ranking of the similarity time series values ​​and the time index values; the output is the correlation coefficient between the rankings of the two sequences. ,in is the Spearman rank correlation coefficient.

[0101] Step 5, if (That is, exhibiting a strong negative correlation and a significant monotonically decreasing trend) and the comprehensive similarity of the time windows at the end of the similarity time series. If the value is below the adaptive threshold, subtract 0.15, where For the first If the overall similarity across all valid time intervals is used, the pair of candidate records is determined to be a feature drift caused by attribute changes, and the pair is kept in the same entity cluster without splitting. Otherwise, they are determined to be different entities, and splitting is performed, assigning the two records to different entity clusters.

[0102] The Spearman correlation coefficient threshold and the upper bound of terminal similarity used for time-series chain alignment can both be adjusted via the system configuration file.

[0103] The enhanced human-machine collaboration confirmation submodule is configured to additionally display the timing change trajectory information for candidate records with insufficient timing chain length and push it to manual review. The enhanced human-machine collaborative confirmation submodule, for candidate record pairs that cannot be automatically determined after time-series chain comparison (i.e., those with fewer than 3 valid time intervals in a single-source time-series record chain are considered insufficient), when pushed to the manual review workbench in Implementation Example 1, adds the attribute change timeline and comprehensive similarity change curve for each record to the original display of detailed information and comprehensive similarity calculation basis for the two records. The attribute change timeline lists the field names and field values ​​before and after the change marker vector component of each single-source time-series record chain node in time axis form. Reviewers perform the judgment based on the time-series trajectory information, and the logic for processing the return of the judgment result is the same as in Implementation Example 1.

[0104] According to the embodiments of this implementation, by adding a time-series marker additional processing module and a single-source time-series record chain generation module to the data access layer, the system retains the time-series evolution context of each record while performing format standardization processing, so that the subsequent entity alignment process has the information basis to perceive the changes of enterprise attributes over time, making up for the deficiency in Embodiment 1 where each record does not carry time-series information when entering the comprehensive similarity calculation process.

[0105] By introducing change marker vectors and cumulative change count vectors into the enhanced enterprise entity feature vector generation submodule, and applying a negative exponential decay factor to the structural attribute similarity calculation in the enhanced entity similarity calculation submodule, the system achieves this. ,in For the first Decay factor for each field dimension, It is a natural constant. The attenuation intensity coefficient, and Records respectively and In their respective single-source time-series record chains, the first The system adaptively perceives the temporal stability of each field dimension at the feature weight level by considering the cumulative number of changes. For fields such as registered capital and address that have undergone multiple historical changes, the high cumulative number of changes causes the decay factor to approach zero, thus reducing the influence of this dimension on the overall similarity judgment. For fields with high historical stability, such as company name, the decay factor is close to 1, maintaining or relatively increasing its influence. Therefore, feature differences caused by normal changes in company attributes will not be erroneously amplified as criteria for determining different entities, solving the problem in Example 1 where the fixed-weight overall similarity calculation cannot cope with the temporal evolution of company attributes.

[0106] By introducing temporal chain alignment operations into the cluster boundary region of the enhanced hierarchical clustering merging submodule and performing Spearman rank correlation coefficient tests on similarity time series, the system can distinguish between "the same entity whose overall similarity decreases at the current moment due to attribute gradual changes" and "different entities that happen to be similar in some attributes": the former's similarity time series shows a significant monotonically decreasing trend. Furthermore, if the overall similarity of the end is below the lower limit of the threshold, the system retains it in the same entity cluster, avoiding incorrect entity splitting due to attribute gradual changes; if the latter does not have the above-mentioned temporal characteristics, the system performs splitting processing, avoiding incorrect merging of different enterprises due to similar historical attributes, thus overcoming the limitation of the static clustering threshold in Example 1 being imperceptible to temporal characteristics at the feature level.

[0107] By adding attribute change timelines and comprehensive similarity change curves for each time window to the manual review workbench of the enhanced human-machine collaborative confirmation submodule, reviewers obtain more sufficient temporal context when performing manual judgment on candidate record pairs with insufficient effective intervals in a single-source temporal record chain. This improves the information sufficiency of manual review decisions. At the same time, after the review results are sent back to the training data pool, the semantic coding model fine-tuning parameters and threshold selector parameters described in Example 1 are continuously optimized, so that the overall alignment capability of the system is continuously enhanced with the accumulation of business.

[0108] The following is the complete application scenario: A provincial-level market supervision data service platform (hereinafter referred to as the "Platform") is responsible for aggregating enterprise information within its jurisdiction. The data sources cover four heterogeneous data sources: the business registration database (data source A), the tax information system (data source B), the judicial announcement platform (data source C), and third-party commercial data service providers (data source D). In the first quarter of 20XX, the Platform performed a full-scale data cleansing pipeline on a batch of enterprise records within its jurisdiction. Taking a technology company (hereinafter referred to as the "Target Company") as a core example, this paper illustrates the data flow process of each main module.

[0109] Data access layer and schema mapping processing module: The four data sources each provide original fields for the target company, with inconsistent field naming and formatting. The schema mapping processing module loads the corresponding JSON mapping configuration files for each data source and aligns the original fields to the standard enterprise information data model. The "Registered Capital (Ten Thousand Yuan)" field in data source A and the "Capital Amount" field in data source B are both aligned to the standard field "Registered Capital" through a many-to-one merging mapping rule; the "Party Name" and "Legal Representative" fields in data source C are aligned to "Company Name" and "Legal Representative" respectively through a one-to-many split mapping; and the "Biz Scope" field in data source D is aligned to "Business Scope" through a one-to-one mapping.

[0110] Table 1. Field Comparison Before and After Schema Mapping

[0111] The format normalization module then performs three types of normalization on the standardized records: converting the full-width number "1000.00" in data source A to half-width "1000.00", removing the thousands separator in data source B and normalizing "10,000,000 yuan" to the value 1000.00 (unified unit of ten thousand yuan), and uniformly converting the date field of each data source (such as "March 15, 2018") to the ISO8601 format "20XX-03-15".

[0112] Structured identifier field validation module: Perform GB32100-2015 verification on the Unified Social Credit Code "91330100MA28ABCD7X" of the target enterprise in data source A. Take the first 17 characters, find the position index of each character according to the character set "0123456789ABCDEFGHJKLMNPQRTUWXY", multiply each character with the weighted factor sequence, sum them, take the modulo 31, and calculate the check code.

[0113] Taking this code as an example, the weighted sum modulo 31 results in 7. The character corresponding to index 24 in the character set is "X", which matches the 18th "X" in the code, so the verification passed. A record from data source D has an incorrect last digit of its unified social credit code and an empty company name field; therefore, it is categorized as a noisy record and will not proceed to the next stage of processing.

[0114] Entity recognition and deduplication module: The enterprise entity feature vector generation submodule generates three types of features for the standardized records of the target enterprise from four data sources. Records from data sources A and C both contain a valid Unified Social Credit Code "91330100MA28ABCD7X". In step one of the entity similarity calculation submodule, precise identifier matching is directly triggered, and the overall similarity is determined to be 1.0, eliminating the need to proceed to steps two and three. Records from data sources B and D both lack the Unified Social Credit Code, requiring calculation in steps two and three.

[0115] Taking a record pair from data source B and data source D as an example, the semantic similarity of their names... Structural attribute similarity Calculate the overall similarity using the default weights: ; With a comprehensive similarity of 0.86 exceeding the adaptive threshold of 0.82, the hierarchical clustering and merging submodule merges the four records into the same enterprise entity cluster.

[0116] Table 2 Entity Similarity Calculation Results

[0117] Multi-source data fusion and conflict resolution module: The dynamic credibility scoring submodule calculates a comprehensive credibility score for four data sources. Taking data source A (business registration database, official government data source) as an example: Authority Level Score Historical accuracy score The last update was 15 days ago. Non-empty field coverage ratio By weight Calculate the overall credibility score:

[0118] Table 3 Overall Reliability Scores for Each Data Source

[0119] The field-level conflict resolution submodule performs conflict resolution on the "Registered Capital" field (numerical): the four data sources provide candidate values ​​of 10 million yuan, 10 million yuan, 9.5 million yuan, and 28 million yuan, respectively. In the interquartile range test... , , The upper boundary is The 28 million yuan provided by data source D exceeded the upper bound and was filtered out as an outlier. The remaining three valid candidate values ​​were then weighted and merged based on their overall credibility score.

[0120] For the "Legal Representative" field (a time-sensitive field), data source A provides "Zhang Mou" (timestamp 15 days from the current date), and data source D provides "Li Mou" (timestamp 3 days from the current date). According to the time-sensitive weighted scoring formula, the candidate value for data source D is... Candidate values ​​for data source A Ultimately, “Zhang” from data source A was selected as the fusion result, and the conflict between the two sources and the basis for the decision were noted in the bloodline record.

[0121] Table 4 Results of Key Field Conflict Resolution

[0122] Intelligent cleaning and quality repair module: The rule engine cleaning submodule performs rule verification on the fused records: the administrative division code of the target enterprise's registered location "330100" passes the compliance check, the unified social credit code passes the verification rule again, and there are no rule items that trigger the repair action.

[0123] The machine learning model cleaning submodule performs typo detection on the company name. The sequence labeling model outputs the confidence score for each character in the name sequence. Since the confidence scores for all characters in the target company name are below 0.70, they are simply marked as having no suspected errors and no correction is triggered. Address parsing is performed on the registered address "No. 188, XX Road, Xihu District, Hangzhou City, Zhejiang Province". The model breaks it down into the provincial level "Zhejiang Province", the municipal level "Hangzhou City", the district / county level "Xihu District", and the street address "No. 188, XX Road". It then matches this with the geocoding dictionary to output the precise administrative division code "330106". Simultaneously, it detects that the original administrative division code "330100" (municipal level code) is inconsistent with the actual address, triggering correction.

[0124] The Isolation Forest algorithm performs outlier detection on the registered capital field of companies in the same industry and region. The target company, with a registered capital of 9.85 million yuan after merging, has an outlier score among the same sample group. If the value is below the marking threshold of 0.75, it will not be marked as an anomaly.

[0125] The missing value filling submodule detected that the "number of employees" field of the target company was missing in all data sources. The first priority multi-source completion had no available information, so it was switched to the second priority: take the median of 47 people from the historical statistics of the same industry (software and information technology services) in the same region as the suggested completion value, mark it as the inferred completion value and attach a confidence level label.

[0126] Data quality assessment module: After the cleaning pipeline is completed, the data quality assessment module performs a five-dimensional scoring on the full dataset of this batch (a total of 3,842 enterprise records, which are deduplicated into 3,617 enterprise entity clusters).

[0127] Table 5. Data Quality Scoring Results for This Batch

[0128] Overall data quality score The quality report is output to the service output layer in both HTML and PDF formats for downstream risk control review and enterprise profiling applications to access.

[0129] Feedback closed-loop interface: Downstream risk control applications on the platform discovered an error in the "Legal Representative" field of the target company's data fusion results during business verification. The value of "Zhang" was incorrect, while the correct value was "Wang". A feedback record containing the company identifier, the incorrect "Legal Representative" field, the incorrect value "Zhang", and the correct value "Wang" was sent back via the feedback loop interface. Upon receiving this feedback, the system added it to the data quality issue log and simultaneously updated the verification count for the "Legal Representative" field in data source A. Add 1 Unchanged, triggered The score is immediately lowered, and the overall credibility score of data source A is updated accordingly in the next 24-hour recalculation cycle. This feedback record is simultaneously added to the machine learning model training data pool. After the training pool reaches the preset batch size, incremental training is triggered, so that subsequent decisions on the resolution of similar time-sensitive fields are adjusted towards more accurate updates.

[0130] Throughout the pipeline, raw, multi-source heterogeneous data is retrieved by the connector plugin in the data access layer and then sequentially undergoes schema mapping, format standardization, and structured identifier verification to transform heterogeneous raw fields into unified standard records. Upon entering the data processing layer, the entity recognition and deduplication module merges target enterprise records scattered across four data sources into a single entity cluster. The multi-source data fusion and conflict resolution module makes verifiable resolution decisions for conflicts in each field based on dynamic credibility scores. The intelligent cleaning and quality repair module performs quality repair in five areas: format compliance, typos, address standardization, numerical anomalies, and missing values. The service output layer generates a quantitative quality report and provides a query interface. Downstream feedback is fed back through the closed-loop interface, continuously updating the data source credibility score and model parameters, forming a complete closed loop for data quality improvement.

[0131] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. An intelligent enterprise information cleaning system based on multi-source data fusion, characterized in that, include: The data access layer is configured to acquire original data from multiple heterogeneous enterprises through the data source connector, and output standardized records that conform to the standard enterprise information data model after schema mapping and format normalization processing. The entity recognition and deduplication module is configured to generate name semantic feature vectors, structural attribute feature vectors, and unique identifiers for standardized enterprise records, calculate the comprehensive similarity, and perform hierarchical clustering to form enterprise entity clusters. The multi-source data fusion and conflict resolution module is configured to perform field-level differential conflict resolution on records from different data sources within the same entity cluster based on the dynamic credibility score of the data source. The intelligent cleaning and quality repair module is configured to perform rule verification and machine learning model detection and correction on the merged enterprise records. The feedback loop interface is configured to receive error records returned by downstream application systems and trigger incremental model training and data source credibility adjustment.

2. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 1, characterized in that, The data access layer also includes a structured identifier field verification module. This module verifies the unified social credit code according to the GB32100-2015 standard, checking the verification code value. The calculation formula is: ； in The first 17 characters The position index of a bit character in the character set For the first The weighting factor corresponding to the position, when The checksum is taken from the character with index 0 in the character set; otherwise, it is taken from the character with index 1. The structured identifier field verification module filters out records where both the unified social credit code and organization code fail verification and the enterprise name field is empty, and classifies them into the noise record pool.

3. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 1, characterized in that, The calculation of comprehensive similarity in the entity recognition and deduplication module includes: Perform exact identifier matching judgment. If there are identifier pairs with the same field name and equal field value in the feature set of two records, the overall similarity is directly determined to be 1.

0. When an exact match cannot be determined, cosine similarity is calculated based on both the name semantic feature vector and the structural attribute feature vector to obtain the name semantic similarity. Similarity with structural attributes ; The overall similarity is calculated using the following formula: ； in To assess overall similarity, and These are the weight coefficients for semantic similarity of names and structural attribute similarity, respectively, satisfying... ; The hierarchical clustering and merging process employs locality-sensitive hashing (LSH) to filter candidate alignment sets of enterprise entity feature vectors. Then, a hierarchical clustering algorithm based on a fully linked strategy is used to merge records with a comprehensive similarity exceeding an adaptive threshold into enterprise entity clusters.

4. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 1, characterized in that, The data source's dynamic credibility score is determined by an authority level score. Historical accuracy score Data update frequency rating and field coverage completeness score The formula for calculating the overall credibility score, which is a weighted composite of four dimensions, is as follows: ； in To assess overall credibility, , , , These are the weight coefficients for the four dimensions, and the sum of each weight is 1. The data update frequency score The calculation formula is: ； in It is a natural constant. This represents the number of days since the last data update of this data source. This is the attenuation coefficient.

5. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 4, characterized in that, The field-level differential conflict resolution includes: For the identifier field, calculate the overall support for each candidate value. ,in To provide candidate values The collection of data sources For data source The overall credibility score is used to select the candidate value with the highest overall support and the authority level score not lower than the set value as the fusion result; For the timeliness field, scores are weighted by timeliness. The candidate value with the highest score is selected as the fusion result, where The difference between the data timestamp of the candidate value and the current time. This is a timeliness weighting coefficient; For numerical fields, an outlier detection method based on interquartile range is used to filter out outlier candidate values, and then the fusion value is calculated by weighting the comprehensive credibility score.

6. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 1, characterized in that, The intelligent cleaning and quality repair module includes: The rules engine cleaning submodule loads a configurable rule base to perform format validation and repair actions on enterprise records; The machine learning model cleaning submodule includes: using a character-level sequence labeling model to perform typo detection and correction on company names; using an address semantic understanding model to perform hierarchical standard parsing on unstructured address text; and using the Isolation Forest algorithm to perform outlier detection on numerical fields, with the Isolation Forest algorithm outputting anomaly scores. ,in The sample to be tested. This represents the total number of samples within the current group. Let be the expected value of the path length among multiple isolated trees. For a sample size of The average path length of a binary search tree at that time; The missing value filling submodule performs missing field filling according to the priority order of multi-source information completion and statistical inference from similar enterprises.

7. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 1, characterized in that, It also includes a data quality assessment module, which quantitatively scores the cleaned dataset from five dimensions: completeness, accuracy, consistency, timeliness, and uniqueness. The formula for calculating the comprehensive data quality score is as follows: ； in To provide a comprehensive data quality score, To score for completeness, For accuracy scoring, For consistency scoring, For timeliness scoring, For uniqueness scoring, , , , , These are the weights for each dimension, and the sum of the weights for each dimension is 1.

8. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 3, characterized in that, The data access layer also includes a time-series marker appending processing module, configured to append data collection timestamps and data source update timestamps to standardized enterprise records; it also includes a single-source time-series record chain generation module, configured to use the data source identifier and the enterprise unique identifier as grouping keys, arrange records of the same data source in different collection batches for the same grouping key in ascending order according to the data source update timestamp to form a single-source time-series record chain, and when inserting a new node, compare each field value with the previous node field by field, and add 1 to the corresponding component of the cumulative change count vector for fields that have changed.

9. The enterprise information intelligent cleaning system based on multi-source data fusion according to claim 8, characterized in that, The entity recognition and deduplication module introduces a time-series confidence decay factor into the structural attribute similarity when calculating the comprehensive similarity. Decay factor for each field dimension The calculation formula is: ； in It is a natural constant. The attenuation intensity coefficient, and These are two records in their respective single-source time-series record chains. The cumulative number of changes for each field dimension; The weighted structural attribute similarity after time-series confidence decay adjustment is: ; in Let be the dimension of the structural attribute feature vector. and These are the first two L2-normalized structural attribute feature vectors of the two records. One component; For candidate record pairs located in the cluster boundary region, a time-series chain comparison operation is performed. After aligning the two single-source time-series record chains according to the time window, the comprehensive similarity of each time interval is calculated to form a similarity time-series sequence. Spearman rank correlation coefficient test is performed on the sequence. When the rank correlation coefficient is lower than the negative correlation threshold and the end comprehensive similarity is lower than the set lower bound, it is determined to be a feature drift caused by attribute change and is retained in the same entity cluster.

10. A method for intelligent cleaning of enterprise information based on multi-source data fusion, used to execute the intelligent cleaning system for enterprise information based on multi-source data fusion as described in any one of claims 1 to 9, characterized in that, include: Step 1: Obtain original data from multiple heterogeneous enterprises through the data source connector, and output standardized records that conform to the standard enterprise information data model after schema mapping and format normalization. Step 2: Generate name semantic feature vectors, structural attribute feature vectors, and unique identifier precise matching features for standardized enterprise records; calculate the comprehensive similarity and perform hierarchical clustering to form enterprise entity clusters; Step 3: Based on the dynamic credibility score of the data source, perform field-level differential conflict resolution on records from different data sources within the same entity cluster; Step 4: Perform rule verification and machine learning model detection and correction on the merged enterprise records; Step 5: Receive error records returned by downstream application systems and trigger incremental model training and data source credibility adjustment.

Images