A multi-source name address big data cleaning, deduplication and standardization processing method

Abstract

The present application relates to big data processing and database data cleaning technical field, especially to a kind of multi-source name address big data cleaning deduplication and standardization processing method. Obtain multi-source name address data record, according to data providing, interface forwarding and source record replication relationship division independent evidence field;Original name field and address field are reversibly cleaned, structured and split and generate name address joint feature;Utilize multi-type blocking key to form candidate duplicate record group, establish consistent edge and conflict edge to constitute name address evidence graph;Through removing the to-be-removed consistent edge in the shortest consistent edge path connecting the conflict edge end point, generate duplicate record cluster;In cluster, support record of the same independent evidence field is counted once, determines standard field value and establishes field evidence chain;After uniqueness, address level path and traceability check, standard name address record is written into standard name address library. The present application can reduce homologous data duplicate counting and conflict record error clustering.

Description

Technical Field

[0001] This invention relates to the field of big data processing and database data cleaning technology, and in particular to a method for cleaning, deduplication, and standardization of multi-source big data. Background Technology

[0002] With the continuous increase in the scale of name and address data in government databases, enterprise databases, publicly available internet databases, and third-party data service platforms, unified big data processing of name and address data from different sources and with different structures has become a common step in database data quality governance. Because different data sources use different field structures, character formats, name representation methods, address hierarchy classification methods, and data update times, the same object may correspond to different name and address field values ​​in different data sources. Furthermore, even within the same data source, there may be duplicate records, invalid records, or records with inconsistent field content.

[0003] Existing multi-source name and address data processing methods typically first analyze the data structure of different data sources, establish the mapping relationship between source fields and standard fields, perform field extraction, format conversion, content splitting and standardization processing on name and address fields, and then perform duplicate checks based on the degree of similarity in name, address or geographical location between different data records. Data records that are determined to be duplicates are then merged, updated or deleted.

[0004] For example, Chinese invention patent application CN112905728A, published on June 4, 2021, discloses an efficient fusion and retrieval system and method for multi-source place name data. The method disclosed in CN112905728A analyzes the internal structure of place name data from different sources, extracts common fields from the data, defines a standardized place name data structure, and converts place name data from different sources into standardized place name data. Subsequently, it performs a duplicate check based on information such as place name and geographical location. If corresponding place name data exists in the place name database, the data is updated; otherwise, the standardized place name data is written into the place name database. The technology disclosed in CN112905728A can realize field mapping, format conversion, duplicate checking, and data fusion of multi-source place name data.

[0005] However, in multi-source big data processing, different data sources do not necessarily correspond to independent data sources. Some data sources may obtain data from the same upstream data source, or form data records with the same or similar content through interface forwarding, batch copying, and redistribution. If the support of candidate field values ​​is directly counted based on the number of data sources or the number of records, multiple records originating from the same initial data may be counted repeatedly as multiple independent supports, thus affecting the determination of standard field values.

[0006] Furthermore, when aggregating duplicate records based on similarity in name, address, or geographic location, if the first and second data records meet the duplication criteria, and the second and third data records also meet the duplication criteria, the three data records may be grouped into the same set of duplicate records due to the propagation of local similarity. However, there may still be mutual exclusion of address entries at the same level, mutual exclusion of parent address paths, or conflicts in numerical location hierarchy between the first and third data records. If aggregation is performed solely based on forward similarity, there is a possibility of incorrectly grouping name and address data records of different objects into the same set of duplicate records, further affecting the determination of the standard name and standard address fields.

[0007] Furthermore, if the correspondence between the standard field values ​​and the original field values ​​is not fully preserved after character format conversion, field splitting, field rearrangement, and determination of the standard field values ​​for the name and address fields, it will be difficult to verify the source of the standard field values, the field cleaning process, and the selection criteria. When there are conflicting records in the duplicate record set, candidate field values ​​from the conflicting records may still participate in the determination of the standard field values.

[0008] Therefore, the main technical problem that existing technologies need to solve is: in the process of multi-source name and address big data processing, when there are data provision, interface forwarding or replication dependencies between different data sources, and when local similarity relationships between records may form transmission clusters, how to reduce the possibility of data from the same initial source being repeatedly counted as multiple independent supports, and name and address data records with address level or digital location conflicts being incorrectly classified into the same duplicate record cluster, and to determine standard field values ​​with traceable basis. Summary of the Invention

[0009] To address the aforementioned technical problems, the present invention aims to provide a method for cleaning, deduplicating, and standardizing multi-source name and address big data. This invention groups data sources that depend on data provision, interface forwarding, or replication into the same independent evidence domain. Multiple supporting records within the same independent evidence domain are counted individually. A name and address evidence graph is constructed using consistent edges and conflicting edges. Duplicate record clusters are split by removing consistent edges from the shortest consistent edge path connecting the two endpoints of conflicting edges. Standard field values ​​are then determined based on these split duplicate record clusters, and a field evidence chain is established from the original field values ​​to the standard field values. This solves the technical problems of duplicate counting of same-source data, erroneous clustering of conflicting records, and the lack of verifiable evidence for standard field values.

[0010] This invention provides a method for cleaning, deduplication, and standardization of multi-source name and address big data, executed by a big data processing platform, including:

[0011] Acquire multi-source name and address data records, divide them into independent evidence domains based on data provision relationships, interface forwarding relationships, and source record replication relationships, and save the source record identifier and original field values;

[0012] The original field values ​​are mapped to a unified name field and a unified address field and reversibly cleaned. The unified name field is split into a core name item, a name qualifying item, and a name alias item. The unified address field is split into a hierarchical address item, a numeric location item, and an unrecognized fragment, generating a joint name-address feature.

[0013] Construct blocking keys based on the joint features of name and address, group records with shared blocking keys into candidate duplicate record groups, establish consistent edges based on independent evidence domain support and name-address compatibility, and establish conflict edges based on address mutual exclusion or digital location subordinate conflict relationship to form a name-address evidence graph.

[0014] For conflicting edges within the same consistent edge connected component, determine the shortest consistent edge path, and determine the consistent edges to be removed according to the number of independent evidence domains supported, the number of common level address entries, and the most recent acquisition time. Remove and recalculate the consistent edge connected component until the two ends of the conflicting edge belong to different consistent edge connected components, and determine the consistent edge connected component as a duplicate record cluster.

[0015] Within a cluster of repeated records, the supporting records of the same independent evidence domain are counted as only one independent support. The standard field value is determined based on the number of independent supports, the continuity of the address hierarchy, and the collection time, and a standard name and address record and field evidence chain are generated.

[0016] Perform uniqueness, address level path and field evidence chain tracing verification on standard name and address records. Verify that the candidate field value does not originate from the record isolated by the conflict edge. Write the standard name and address records that pass the verification into the standard name and address database, and write the standard name and address records that fail the verification into the abnormal record set.

[0017] In some implementations, the determination of independent evidence domains includes: taking each data source as a source node, establishing data providing edges, interface forwarding edges, and source record copying edges between upstream and downstream source nodes according to the data flow direction, forming a data source dependency graph; forming dependency closure groups by grouping mutually reachable source nodes in the data source dependency graph, and shrinking each dependency closure group into a corresponding equivalent source node; determining the source node or equivalent source node without any incoming edges in the shrunk data source dependency graph as the initial source node; and classifying source nodes that can be traced back to the same initial source node along the reverse direction of the data providing edge, interface forwarding edge, or source record copying edge into the same independent evidence domain.

[0018] In some implementations, after performing reversible cleaning and generating joint name-address features, when there is no explicit source record replication relationship between the two source nodes, source record content fingerprints for each multi-source name-address data record are generated based on the name core item, hierarchical address item, numeric location item, and original field values, respectively. When each source record content fingerprint in the target acquisition batch of the second source node is the same as a source record content fingerprint in the previous acquisition batch of the first source node, and the value sets of the name core item, hierarchical address item, and numeric location item in the target acquisition batch of the second source node are respectively included in the corresponding value sets in the previous acquisition batch of the first source node, a source record replication edge from the first source node to the second source node is established. The data source dependency graph is updated based on the newly added source record replication edge, and the independent evidence domain containing the first and second source nodes is redefined.

[0019] In some implementations, reversible cleaning includes: for each original field value, recording the field value before cleaning, the field value after cleaning, the cleaning operation identifier, and the cleaning rule version according to the execution order of the cleaning operation to form a field transformation chain; the cleaning operation identifier includes at least one of character encoding conversion, full-width / half-width conversion, punctuation normalization, null value placeholder removal, number format unification, field splitting, and field rearrangement; performing reverse restoration verification on the field transformation chain, using the cleaned field value that can be restored to the corresponding original field value after reverse restoration verification to generate a name-address joint feature, marking the cleaned field value that fails the reverse restoration verification as a transformation abnormal field value, and stopping the establishment of consistent edges using the transformation abnormal field value.

[0020] In some implementations, the construction of the blocking key includes: matching hierarchical address entries with a pre-established standard address hierarchy tree, and determining the effective address path with the most hierarchical continuity and the largest number of covered address segments from the matching results; combining the name core entry with the last-level address entry of the effective address path to form a first blocking key, combining the name core entry, the numeric location entry, and the parent address entry corresponding to the numeric location entry to form a second blocking key, generating an unidentified segment content fingerprint based on the unidentified segment, and combining the name alias entry, the effective address path, and the unidentified segment content fingerprint to form a third blocking key; writing multi-source name-address data records with the same first blocking key, the same second blocking key, or the same third blocking key into the corresponding blocking key buckets; merging blocking key buckets with overlapping records into blocking key bucket closures based on the source record identifier; retaining two blocking key buckets with corresponding parent address paths that are mutually exclusive in the standard address hierarchy tree in different blocking key bucket closures, and determining each blocking key bucket closure as a candidate duplicate record group.

[0021] In some implementations, the establishment of consistent edges and conflicting edges includes: establishing a consistent edge between two multi-source address data records when the name core entries of two multi-source address data records are the same, or when the name alias entry of one multi-source address data record has a one-to-one alias relationship with the name core entry of another multi-source address data record, and the effective address paths corresponding to the two multi-source address data records can be connected to the same parent-child path in the standard address hierarchy tree; and establishing a consistent edge when the name core entry, name alias entry, effective address path, and digital location entry on which the consistent edge is based consist of multi-source address data from at least two independent evidence domains. If the records support each other, or if two multi-source address data records belong to consecutive acquisition versions of the same source record identifier and have the same corresponding digital location item, the corresponding consistent edge is marked as a strengthened consistent edge; when the same level address items of two multi-source address data records correspond to mutually exclusive nodes in the standard address hierarchy tree, or the same digital location item belongs to mutually exclusive parent address paths, a conflict edge is established between the two multi-source address data records; each consistent edge and each conflict edge is associated with the corresponding source record identifier, independent evidence field identifier, blocking key type, field conversion chain, and acquisition time to form an edge evidence set.

[0022] In some implementations, determining the shortest consistent edge path includes: extracting an initial consistent edge connected component formed by consistent edges from the name-address evidence graph; determining target conflict edges whose endpoints are both located within the same initial consistent edge connected component; determining all shortest consistent edge paths connecting the endpoints of each target conflict edge; merging all shortest consistent edge paths corresponding to the same target conflict edge; and determining the union of the merged results corresponding to each target conflict edge as a conflict subgraph; and counting the number of independent evidence domains supported, the number of common level address entries, the most recent acquisition time, and the number of times they are covered by the shortest consistent edge paths corresponding to different target conflict edges in each consistent edge of the conflict subgraph.

[0023] In some implementations, removing a consistent edge to be removed includes: identifying consistent edges among the strengthened consistent edges that are supported by at least two independent evidence domains and whose corresponding digital location items are the same as anchored consistent edges; sorting the consistent edges in the conflict subgraph according to whether they are anchored consistent edges, the number of independent evidence domains supporting them, the number of common hierarchical address items, the most recent acquisition time, and the number of common coverages, wherein non-anchored consistent edges are ranked before anchored consistent edges, the number of independent evidence domains supporting them and the number of common hierarchical address items are ranked in ascending order, the most recent acquisition time is ranked in ascending order, and the number of common coverages is ranked in descending order; identifying the consistent edge at the top of the ranking as a consistent edge to be removed; after removing the consistent edge to be removed, recalculating the connected components of the consistent edges, and redetermining the target conflict edges whose endpoints are still located in the same connected component of the consistent edges, until the two endpoints of each target conflict edge are located in different connected components of the consistent edges.

[0024] In some implementations, after obtaining the duplicate record clusters, the method further includes: verifying whether there are conflicting edges within each duplicate record cluster whose endpoints are both located within the duplicate record cluster, and identifying the existing conflicting edges as internal conflicting edges; identifying multi-source name-address data records that participate in at least one anchored consistent edge as anchored records; for each multi-source name-address data record that does not belong to an anchored record, verifying whether there is a connection path consisting of consistent edges between the multi-source name-address data record and at least one anchored record, and verifying whether each consistent edge in the connection path has an edge evidence set that can be traced back to the original field value along the corresponding field transformation chain; removing multi-source name-address data records that serve as endpoints of internal conflicting edges, multi-source name-address data records that fail the connection path verification, and multi-source name-address data records that fail the edge evidence set verification from the corresponding duplicate record clusters and writing them into the abnormal record set, and using the duplicate record clusters after the records have been removed to determine the standard field values.

[0025] In some implementations, the field evidence chain adopts a directed acyclic structure and includes source field value nodes, cleansing field value nodes, candidate field value nodes, standard field value nodes, and duplicate record cluster version nodes. A transformation edge is established between the source field value node and the cleansing field value node, a support edge is established between the cleansing field value node and the candidate field value node, an exclusion edge is established between the cleansing field value node (which originates from multi-source name-address data records isolated by internal conflict edges and removed from the corresponding duplicate record cluster) and the corresponding candidate field value node, a selection edge is established between the candidate field value node and the standard field value node, and version association edges are established between the candidate field value node, the standard field value node, and the corresponding duplicate record cluster version node, respectively. Each standard field value node can be traced back to the source field value node corresponding to each independent support along the reverse direction of the selection edge, support edge, and transformation edge, and the candidate field value node connected to the standard field value node through the selection edge is not connected to the corresponding cleansing field value node through the exclusion edge.

[0026] In some implementations, determining the standard field value corresponding to the unified address field includes: constructing candidate standard address paths using only hierarchical address items and numeric location items within the same duplicate record cluster that are not connected to the corresponding candidate field value node via exclusion edges; verifying the parent-child relationship of each candidate standard address path in the standard address hierarchy tree, and determining the target standard address path from each candidate standard address path in descending order of the number of consecutive address levels and the number of independent supports; when the target standard address path lacks a parent address item, and the corresponding lower-level address item corresponds to only one parent address item in the standard address hierarchy tree, adding the parent address item to the target standard address path, and establishing a completion basis edge between the candidate field value node corresponding to the parent address item and the candidate field value node corresponding to the lower-level address item; when the lower-level address item corresponds to at least two parent address items in the standard address hierarchy tree, marking the lower-level address item as a parent ambiguous address item, and re-performing conflict edge establishment and consistent edge connectivity component splitting on the duplicate record cluster containing the parent ambiguous address item.

[0027] In some implementations, determining the standard field value corresponding to the unified name field includes: extracting the core name item and alias name item from multi-source name and address data records that correspond to the same target standard address path and are not internal conflict edge endpoints; establishing an alias mapping between the core name item and the alias name item when a core name item and an alias name item are supported by at least two independent evidence fields, or when the core name item and the alias name item belong to consecutively acquired versions of the same source record identifier and the corresponding digital location items are the same; determining the standard core name item in descending order of the number of independent supports, and writing the remaining core name items and alias name items connected to the standard core name item through alias mapping into the alias field of the standard name and address record; when the two endpoints of the newly added conflict edge respectively contain the standard core name item and the alias name item that has an alias mapping with the standard core name item, canceling the corresponding alias mapping, and restoring the core name item and alias name field values ​​corresponding to the alias mapping before it was established along the field evidence chain.

[0028] In some implementations, when performing incremental processing on newly added, modified, or deleted multi-source address data records, the process includes: for newly added or modified multi-source address data records, determining the corresponding blocking key bucket and the blocking key bucket closure that overlaps with the blocking key bucket; for deleted multi-source address data records, determining the corresponding blocking key bucket and the blocking key bucket closure based on the blocking key in the previous valid version; determining the multi-source address data records in the blocking key bucket closure as initial impact records; adding multi-source address data records that can be reached from the initial impact records via consistent edges to the impact record set; and adding multi-source address data records that can be reached from the impact record set via conflicting edges. The multi-source name and address data records connected by the records are added to the affected record set, and the multi-source name and address data records contained in the shortest consistent edge path connecting the two ends of the conflicting edge are added to the affected record set; the candidate field value nodes and standard field value nodes connected by the version association edge of the duplicate record cluster version node corresponding to the affected record set are added to the affected node set, and the affected record set and the affected node set form an incremental affected closure; only the candidate duplicate record group generation, name and address evidence graph construction, duplicate record cluster splitting and standard field value determination are re-executed for the incremental affected closure, while keeping the duplicate record clusters, standard field values ​​and field evidence chains outside the incremental affected closure unchanged.

[0029] In some implementations, version control of incremental impact closures includes: establishing an incremental processing version and recording changes in newly added consistent edges, deleted consistent edges, newly added conflicting edges, deleted conflicting edges, duplicate record cluster members, standard field values, and field evidence chains; performing uniqueness checks and address-level path checks on the updated standard name-address records, and verifying that each updated standard field value can be traced back to its corresponding original field value along the field evidence chain; when all checks pass, atomically committing the consistent edges, conflicting edges, duplicate record clusters, standard field values, and field evidence chains corresponding to the incremental processing version; when any check fails, deleting the newly added consistent edges, conflicting edges, selection edges, and duplicate record cluster version nodes of the incremental processing version, restoring the deleted consistent edges and conflicting edges of the incremental processing version, and restoring the corresponding duplicate record clusters and standard field values ​​according to the field evidence chain corresponding to the previous incremental processing version.

[0030] In some implementations, the big data processing platform includes multiple data processing nodes. These nodes receive corresponding multi-source name-address data records based on the stable hash value of the blocking key. For multi-source name-address data records with at least two blocking keys and assigned to different data processing nodes, a boundary record index is established, containing source record identifiers, blocking keys, and data processing node identifiers. Before constructing the name-address evidence graph, candidate duplicate record groups containing the same source record identifier are merged according to the boundary record index. After each data processing node completes the consistent edge connectivity component splitting of the corresponding name-address evidence graph, it determines the cross-node conflict edges connecting the duplicate record clusters formed by different data processing nodes based on the boundary record index. When the two endpoints of each cross-node conflict edge are located in different duplicate record clusters, and the duplicate record cluster versions corresponding to each data processing node are consistent, a standard name-address record and field evidence chain are submitted.

[0031] Compared with existing technologies, the above technical solution has the following advantages:

[0032] 1. This invention divides independent evidence domains according to data provision relationships, interface forwarding relationships, and source record replication relationships, and counts multiple supporting records for the same candidate field value in the same independent evidence domain as one independent support. This can reduce duplicate counts formed after the same initial data is forwarded, replicated, or repeatedly published by multiple data sources, and reduce the possibility of non-independent data sources having a duplicate impact on the determination result of standard field values, thereby improving the reliability of deduplication and standardization processing of multi-source name address data.

[0033] 2. This invention establishes consistent edges based on name-address compatibility and conflict edges based on mutual exclusion of address items at the same level, mutual exclusion of parent address paths, and conflict of digital location subordinate relationships. This enables positive similarity evidence and negative mutual exclusion evidence to work together in the same name-address evidence map to identify duplicate records. It can identify conflict situations such as different addresses with the same name or different parent address paths with the same address, which are difficult to find based on name or local address similarity alone.

[0034] 3. In the case where the two ends of a conflicting edge are located in the same consistent edge connected component, this invention determines the shortest consistent edge path connecting the two ends of the conflicting edge, and determines the consistent edges to be removed according to the number of independent evidence domains supported, the number of common level address items, and the acquisition time, until the two ends of the conflicting edge belong to different consistent edge connected components. This can suppress erroneous clustering caused by the transmission of local similarity relationships and reduce the possibility of different name and address objects being incorrectly classified into the same duplicate record cluster.

[0035] 4. After completing the conflict constraint clustering, this invention determines the standard field value based on the number of independent supports, the continuity of address hierarchy, and the acquisition time, and performs standard address hierarchy tree verification on the candidate standard address paths. This can reduce the interference of conflict records, address hierarchy break records, and duplicate source records on the standard name field value and the standard address field value, and improve the integrity and consistency of the standard name and address records.

[0036] 5. This invention saves the original field values, performs reversible cleaning on the name and address fields, and establishes a field evidence chain consisting of source field value nodes, cleaned field value nodes, candidate field value nodes, standard field value nodes, and duplicate record cluster version nodes. This enables the standard field value to be traced back to the corresponding original field value and field conversion process, facilitating source verification, conversion process verification, and version verification of the standardized processing results.

[0037] 6. This invention determines the blocking key bucket and its corresponding blocking key bucket closure based on the addition, modification or deletion of records, and forms an incremental influence closure along consistent edges, conflicting edges, shortest consistent edge paths and version associations. Only the data within the incremental influence closure is re-executed for candidate generation, graph construction, clustering and standard field value determination, which can reduce the repeated reading of irrelevant data, graph calculation and storage writing, and reduce processor usage, running memory usage and cross-node data transmission volume during incremental update process.

[0038] 7. This invention distributes multi-source name-address data records to multiple data processing nodes according to the stable hash value of the blocking key, and uses boundary record index to merge cross-node candidate duplicate record groups and verify cross-node conflict edges. It can achieve multi-node parallel processing while maintaining the version consistency of duplicate record clusters, and is suitable for cleaning, deduplication and standardization of large-scale multi-source name-address data. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the hardware architecture of the multi-source name and address big data cleaning and processing platform of the present invention.

[0041] Figure 2 This is a schematic diagram of the functional module structure of the multi-source name and address big data cleaning and processing platform of the present invention.

[0042] Figure 3 This is a flowchart illustrating the multi-source address big data cleaning, deduplication, and standardization method of the present invention.

[0043] Figure 4 This is a flowchart illustrating the data source dependency graph construction and independent evidence domain identification method of the present invention.

[0044] Figure 5 This is a flowchart illustrating the conflict resolution method based on the name-address evidence map of the present invention.

[0045] Figure 6 This is a flowchart illustrating the incremental impact closure processing and version control method of the present invention.

[0046] Figure 7 This is a schematic diagram of the topological changes during the data source dependency graph shrinkage and independent evidence domain partitioning process of the present invention.

[0047] Figure 8 This is a schematic diagram showing the state changes of the address evidence map before and after splitting in this invention.

[0048] Figure 9 This is a schematic diagram of the directed acyclic graph structure of the field evidence chain of the present invention.

[0049] Figure 10 This is a schematic diagram illustrating the distribution mapping between the blocking key, the blocking key bucket, and the data processing node in this invention.

[0050] Figure 11 This diagram illustrates the comparison of processing results and update performance for different address data processing methods. Figure 11 chart 'a' shows a comparison of the standard field values ​​processed using different address data processing methods. Figure 11 b is a comparison chart of the processing performance of full update and incremental update affecting closure update. Detailed Implementation

[0052] The following detailed description, with reference to the accompanying drawings, illustrates a method for cleaning, deduplicating, and standardizing multi-source address big data. The implementation details describe the complete processing of multi-source address data records, from receipt, field cleaning, generation of candidate duplicate records, splitting of duplicate record clusters, determination of standard field values, to result writing. This enables those skilled in the art to implement this method based on the disclosed data structure, processing conditions, and execution order. The number of data sources, data processing node configurations, number of records, character content, and test data listed below are specific embodiments and do not constitute limitations on data sources, data scale, or hardware configuration.

[0053] Example 1

[0054] like Figure 1 As shown, the big data processing platform used in this embodiment includes a multi-source database, a data receiving interface, a batch buffer, multiple data processing nodes, a distributed task coordination component, a boundary record index storage module, a standard name and address database, and an exception record set. The multi-source database stores name and address data from different data providers or different business systems. The data receiving interface connects to each multi-source database and writes the received multi-source name and address data records into the batch buffer. The data receiving interface can be a database connection interface, an application programming interface, a message queue interface, or a file transfer interface.

[0055] Each data processing node includes a processor, running memory, a local data cache, and a task execution process. The distributed task coordination component is used to assign blocking key processing tasks to each data processing node, maintain task status, and coordinate the merging of candidate duplicate record groups across data processing nodes. The standard name address library stores verified standard name address records, while the abnormal record set stores data that failed structure verification, reversible cleaning verification, duplicate record cluster verification, or field evidence chain verification. The boundary record index storage module stores the source record identifier, blocking key, and data processing node identifier across data processing nodes.

[0056] In one specific deployment configuration, the big data processing platform comprises three data processing nodes, each configured with a 16-core processor, 64 GB of RAM, and 2 TB of solid-state storage. The data receiving interface receives multi-source name-address data records in batches, with each batch assigned a unique batch identifier. Each multi-source name-address data record includes at least the data source identifier, source record identifier, batch identifier, collection time, original name field value, and original address field value.

[0057] In a feasible data table structure, the source record table includes at least the following fields: source record identifier, data source identifier, batch identifier, collection time, original name, original address, current processing status, and current record version. The source record identifier remains unique within the same data source; the data source identifier is used to associate with the data source dependency graph; the collection time uses a unified time zone and is converted to a timestamp for storage; the current processing status indicates whether it is pending cleaning, cleaned, clustered, stored, or in an abnormal state; and the current record version supports incremental changes and deletions.

[0058] like Figure 2 and Figure 3 As shown, after receiving multi-source name and address data records, the big data processing platform sequentially performs data source dependency identification, reversible cleaning and name and address joint feature generation, blocking key and name and address evidence graph construction, conflict splitting and duplicate record cluster generation, standard field value and field evidence chain generation, and verification and database entry processing.

[0059] For ease of explanation, this embodiment selects a portion of multi-source address data records from eight data sources for processing. The eight data sources are respectively identified by data source identifiers SA, SB, SC, SD, SE, SF, SG, and SH, and the received portion of the original multi-source address data records are shown in Table 1.

[0060] Table 1 Original Multi-Source Name and Address Data Record Table

[0061] R001 SA B20260601-01 2026 / 6 / 1 9:00 Beijing ** Technology Co., Ltd. Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing R002 SB B20260601-02 2026 / 6 / 1 9:05 Beijing ** Technology Co., Ltd. Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing R003 SC B20260601-03 2026 / 6 / 1 9:08 Beijing ** Technology Co., Ltd. Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing R004 SD B20260602-01 2026 / 6 / 2 10:00 **Technology (Beijing) Co., Ltd. Room 1201, Building A, No. 88 Jianguo Road, Chaoyang District, Beijing R005 SE B20260603-01 2026 / 6 / 3 11:00 Beijing ** Technology Co., Ltd. Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing R006 SF B20260604-01 2026 / 6 / 4 12:00 **science and technology Room *, Building *, No. ** Jianguo Road, Beijing R007 SG B20260605-01 2026 / 6 / 5 14:00 Beijing ** Technology Co., Ltd. Room *, Building *, No. ** Jianguo Road, Haidian District, Beijing R008 SH B20260606-01 2026 / 6 / 6 14:30 **Technology Beijing Company** Room *, Building *, No. ** Jianguo Road, Haidian District, Beijing

[0062] As shown in Table 1, the original name and original address field values ​​of source records R001, R002, and R003 are basically the same, but they come from three different data sources. Source records R004 and R005 use different name or address representation methods. Source record R006 lacks a district-level address entry. Source records R007 and R008 contain addresses from Haidian District, which have a mutual exclusion relationship with the Chaoyang District addresses contained in source records R001, R004, and R005.

[0063] Upon receiving each batch of data, the data receiving interface first verifies the completeness of the source record identifier, data source identifier, collection time, and original field values. Records with empty data source identifiers or source record identifiers are not included in subsequent name-address evidence graph processing but are instead written to the abnormal record set. Records with empty original name field values ​​but complete original address field values ​​can continue to undergo address field cleaning, but name compatibility is not used as a basis when establishing consistent edges. Records with both empty original name field values ​​and empty original address field values ​​are written to the abnormal record set.

[0064] After completing the basic field validation, the big data processing platform obtains the data provision relationships, interface forwarding relationships, and source record replication relationships between various data sources. Data provision relationships can be obtained from data interface configuration, data exchange protocols, or data provider identifiers; interface forwarding relationships can be obtained from interface call logs, message queue forwarding logs, or data exchange link configurations; and source record replication relationships can be obtained from upstream source record identifiers, replication task identifiers, or batch replication logs.

[0065] In this embodiment, data source SA provides name and address data to data source SB. There is a bidirectional batch synchronization relationship between data source SB and data source SC. Data source SF provides basic name field and numeric location field to data source SG. After receiving the data, data source SG supplements the area-level address field. Data source SD, data source SE and data source SH collect name and address data independently.

[0066] In the data source dependency table, each data source dependency stores at least the upstream data source identifier, downstream data source identifier, relationship type, relationship effective time, and relationship evidence. Relationship types include data provision relationships, interface forwarding relationships, and source record replication relationships. Relationship evidence can be an interface configuration identifier, a forwarding log identifier, a replication task identifier, or a relationship identifier inferred from the fingerprint of the source record content.

[0067] like Figure 4 and Figure 7 As shown, the big data processing platform uses each data source as a source node and establishes data provision edges, interface forwarding edges, and source record replication edges between the corresponding source nodes according to the data flow direction, thus forming a data source dependency graph. A data provision edge is established between data source SA and data source SB, interface forwarding edges are established between data source SB and data source SC respectively, and a data provision edge or source record replication edge is established between data source SF and data source SG.

[0068] For mutually reachable source nodes in the data source dependency graph, the big data processing platform groups these mutually reachable source nodes into dependency closure groups, and then shrinks each dependency closure group into an equivalent source node. In practice, either the Tarjan strongly connected component algorithm or the Kosaraju strongly connected component algorithm can be used to determine mutually reachable source nodes.

[0069] When using Tarjan's strongly connected component algorithm, an access sequence number and a low-link value are assigned to each source node, and a depth-first traversal is performed starting from the unvisited source nodes. When the current source node accesses an adjacent source node via a data-providing edge, an interface-forwarding edge, or a source record-copying edge, the low-link value of the current source node is updated. When the access sequence number of the current source node equals the low-link value, source nodes starting from the current source node are popped from the stack and formed into a dependency closure group. This process belongs to the determination process of strongly connected components in a directed graph, and those skilled in the art can directly implement it using existing graph computing libraries.

[0070] In this embodiment, data source SB and data source SC are mutually reachable; therefore, data source SB and data source SC form a dependency closure group. After shrinking the dependency closure group, the source node or equivalent source node in the data source dependency graph that has no incoming edges is determined as the initial source node.

[0071] The big data processing platform traces each source node in the reverse direction of the data providing edge, interface forwarding edge, or source record copying edge. Source nodes that can be traced back to the same initial source node are grouped into the same independent evidence domain. In this embodiment, data source SA, data source SB, and data source SC are grouped into the first independent evidence domain, data source SD into the second independent evidence domain, data source SE into the third independent evidence domain, data source SF and data source SG into the fourth independent evidence domain, and data source SH into the fifth independent evidence domain.

[0072] The independent evidence field table includes at least an independent evidence field identifier, an initial source node identifier, a set of member source nodes, a generation time, and a generation basis. Each source record identifier in the source record table is further associated with a corresponding independent evidence field identifier.

[0073] When a source node can be traced back to more than two initial source nodes along different data paths, the big data processing platform determines the initial source of each multi-source address data record based on the data provision edge, interface forwarding edge, or source record replication edge corresponding to the specific source record identifier. For example, when the same data source simultaneously receives data from two upstream data sources, the initial source of the specific record can be determined based on the upstream source record identifier, interface call identifier, or replication task identifier. When the initial source cannot be uniquely determined based on the specific source record path, the corresponding evidence branches are retained separately, multiple initial sources are not merged into the same independent evidence domain, and the source record identifiers with ambiguous source attribution are written into the abnormal record set.

[0074] Therefore, although source records R001, R002, and R003 originate from three different data sources, all three can be traced back to data source SA. Consequently, when the number of supports for the same candidate field value is subsequently counted, only one independent support is formed. Similarly, although source records R006 and R007 have different data source identifiers, their respective data sources both belong to the fourth independent evidence domain. When both support the same candidate field value, only one independent support is formed.

[0075] After performing reversible cleaning and generating joint name-address features, when there is no explicit source record replication relationship between two source nodes, the big data processing platform generates source record content fingerprints based on the corresponding multi-source name-address data records. Specifically, the core name item, the hierarchical address items arranged in address hierarchy order, the numeric location items arranged in the order of house number, building number, unit number, and room number, and the original field values ​​are all converted to UTF-8 character encoding, and then concatenated in a fixed order of the core name item, hierarchical address items, numeric location items, and original field values. Preset delimiters that will not appear in ordinary name-address text are added between each field, such as the unit delimiter with hexadecimal 1F character encoding. Then, the SHA-256 algorithm is used to calculate the source record content fingerprint.

[0076] The input string for the source record content fingerprint can be concatenated in the following order: "Name Core Item - Hierarchical Address Item - Numeric Location Item - Original Name Field Value - Original Address Field Value". If any field is empty, a preset null value is written in the corresponding position to avoid generating the same concatenation string for different fields with missing values.

[0077] For each target collection batch of the second source node, the big data processing platform searches for identical source record content fingerprints in the previous collection batches of the first source node. When the source record content fingerprint of each source record in the target collection batch of the second source node can be found in the previous collection batches of the first source node, and the value sets of the name core item, the hierarchical address item, and the number positioning item in the target collection batch of the second source node are respectively included in the corresponding value sets in the previous collection batches of the first source node, a source record replication edge is established from the first source node to the second source node.

[0078] For example, all source record content fingerprints in the target acquisition batch of data source SC can be found in the same source record content fingerprints in the previous acquisition batch of data source SB. Furthermore, the target acquisition batch of data source SC does not have a name core item, hierarchical address item, or numerical location item that is different from the previous acquisition batch of data source SB. Therefore, a source record replication edge is established between data source SB and data source SC, and an independent evidence domain containing data source SB and data source SC is redefined.

[0079] If the target acquisition batch contains source record fingerprints that cannot be found in previous acquisition batches, or if the target acquisition batch contains name core items, hierarchical address items, or numeric location items that do not exist in previous acquisition batches, a source record replication edge will not be established based on the corresponding acquisition batch. This avoids two data sources being identified as having a replication relationship simply because some records are the same.

[0080] After the independent evidence domain is determined, the big data processing platform saves the source record identifier and original field value of each multi-source name and address data record, and maps the original name field to the unified name field and the original address field to the unified address field according to the source field mapping configuration.

[0081] The source field mapping configuration must at least record the data source identifier, source field path, target field type, character encoding method, field delimiter method, null value representation method, and cleansing rule version. For example, the name field path of data source SA is "company_name", and the address field path is "registered_address"; the name field path of data source SD is "entity.fullName", and the address field path is "entity.location.text". The data receiving interface reads the field content according to the corresponding source field path and writes it to the unified name field or unified address field.

[0082] When performing reversible cleaning on the Uniform Name and Uniform Address fields, the big data processing platform records the field value before cleaning, the field value after cleaning, the cleaning operation identifier, the cleaning rule version, the original content of the replaced character, the position of the replaced character, and the field split boundary for each original field value in the order of the cleaning operation execution, thus forming a field transformation chain.

[0083] The field transformation linked list includes at least the transformation chain identifier, source record identifier, field type, execution sequence number, field value before cleaning, field value after cleaning, cleaning operation identifier, cleaning rule version, reverse operation identifier, and reverse operation parameters. Each cleaning operation is linked according to its execution sequence number, and the field value after cleaning in the previous cleaning operation is the field value before cleaning in the next cleaning operation.

[0084] Taking the original address field value of source record R001, “No. **, Building / Room *, Jianguo Road, Chaoyang District, Beijing”, as an example, the big data processing platform first performs a full-width / half-width character conversion, converting the full-width character “A” to the half-width character “A”. The platform records the original character “A”, the converted character “A”, the original character position, and the cleaning rule version V1.2 in the field conversion chain. Then, it performs number format unification and address hierarchy splitting. After reversible cleaning, the unified address field is “No. **, Building / Room *, Jianguo Road, Chaoyang District, Beijing”.

[0085] During the reverse restoration verification, the big data processing platform calls the corresponding inverse operations sequentially in reverse order of the field conversion chain to restore the half-width character "A" to the full-width character "A" at its original position, and reconnects the address fragments according to the field split boundaries of the record. When the restored field value matches the original address field value character by character, the reverse restoration verification is considered successful.

[0086] For standardized administrative division operations, such as converting "Beijing Chaoyang District" to "Beijing Chaoyang District", the field conversion chain records the original administrative division character "Beijing", the standard administrative division character "Beijing", the corresponding character position and the version of the administrative division mapping table. During reverse restoration, the original administrative division character is restored based on the corresponding administrative division mapping item.

[0087] For field rearrangement operations, the field transformation chain saves the original sequence number of each character segment before rearrangement and the sequence number after rearrangement; for punctuation normalization, null value placeholder clearing, or special character escaping operations, the field transformation chain saves the original content and character position of the replaced or deleted character.

[0088] For example, when the original name field value is "A&B Company", the special character "&" can be escaped to a predetermined character, and the cleaning operation identifier, character position and original character can be recorded in the field conversion chain. During reverse restoration, the predetermined character can be restored to "&" according to the corresponding reverse operation.

[0089] Results obtained through truncated hashing, direct deletion without preserving character content, or other processing methods that prevent the recovery of original field values ​​based on the field transformation chain are not considered as successful reversible cleaning. Cleaned field values ​​that fail the reverse recovery verification are marked as transformation aberration field values. Consistent edges are not established using transformation aberration field values, and the transformation aberration field values, source record identifiers, and corresponding cleaning rule versions are written to the aberration record set.

[0090] The conversion chains of some fields in this embodiment are shown in Table 2.

[0091] Table 2 Field Transformation Chain Example

[0092] R001 Address field Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing Full-width and half-width conversion V1.2 Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing pass R004 Name field **Technology (Beijing) Co., Ltd. Parentheses unification and field splitting V1.3 **Technology (Beijing) Co., Ltd. pass R005 Address field Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing Standardization of administrative divisions V2.1 Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing pass R006 Address field Room *, Building *, No. ** Jianguo Road, Beijing Address hierarchy splitting V2.1 Room 1201, Building A, No. 88 Jianguo Road, Beijing pass

[0093] After completing the reversible cleaning, the big data processing platform splits the unified name field into name core items, name qualification items, and name alias items, and splits the unified address field into hierarchical address items, numeric location items, and unrecognized fragments.

[0094] When splitting the name field, an organizational form dictionary, an administrative region dictionary, and a name alias table can be pre-built. The organizational form dictionary includes organizational forms such as "Limited Company", "Joint-Stock Company", "Branch Company", and "Center"; the administrative region dictionary includes regional restriction characters such as "Beijing" and "Shanghai"; and the name alias table stores the correspondence between the verified abbreviations, historical names, and standard names.

[0095] The big data processing platform first identifies the organizational form and administrative region restriction characters in the unified name field, determines the remaining character fragments that can represent the main body of the name as the core name item, determines the identified organizational form, administrative region or branch attributes as name restriction items, and saves the complete historical name, abbreviation or name value with different expression order as name alias item.

[0096] For example, "Beijing ** Technology Co., Ltd." is split into the core name item "** Technology" and the name limiting item "Beijing, Co., Ltd."; "** Technology (Beijing) Co., Ltd." is split into the core name item "** Technology", the name limiting item "Beijing, Co., Ltd." and the name alias item "** Technology (Beijing) Co., Ltd.".

[0097] When splitting a unified address field, the big data processing platform divides the unified address field into a sequence of address fragments according to character order. Then, it matches these address fragment sequences with the node names and aliases in the standard address hierarchy tree. The matching can employ a dictionary-longest matching method, selecting the longest character fragment that matches a node in the standard address hierarchy tree from the current character position, and continuing to match subsequent characters according to the parent-child hierarchy.

[0098] Address segments that can be connected according to the parent-child relationship of the standard address hierarchy tree are identified as hierarchical address entries. Numeric segments containing location identifiers such as "number," "building," "seat," "unit," and "room" are identified as numeric location entries and saved separately according to door number, building number, unit number, and room number. Character segments that are not included in the hierarchical address entries and are not identified as numeric location entries are saved as unrecognized segments.

[0099] The field splitting results of the listed multi-source name address data records are shown in Table 3.

[0100] Table 3 Examples of Field Splitting and Name-Address Joint Features

[0101] R001 **science and technology Beijing, Limited Company none Beijing / Chaoyang District / Jianguo Road **Section / Room** none R002 **science and technology Beijing, Limited Company none Beijing / Chaoyang District / Jianguo Road **Section / Room** none R003 **science and technology Beijing, Limited Company none Beijing / Chaoyang District / Jianguo Road **Section / Room** none R004 **science and technology Beijing, Limited Company **Technology (Beijing) Co., Ltd. Beijing / Chaoyang District / Jianguo Road **Section / Room** none R005 **science and technology Beijing, Limited Company none Beijing / Chaoyang District / Jianguo Road **Section / Room** none R006 **science and technology none **science and technology Beijing / Jianguo Road **Section / Room** none R007 **science and technology Beijing, Limited Company none Beijing / Haidian District / Jianguo Road **Section / Room** none R008 **science and technology Beijing, Company **Technology Beijing Company** Beijing / Haidian District / Jianguo Road **Section / Room** none

[0102] The big data processing platform generates a name-address joint feature based on the core name item, name limitation item, name alias item, hierarchical address item, digital location item, unidentified fragment content fingerprint, data source identifier, and independent evidence domain identifier, and then associates and saves the name-address joint feature with the corresponding source record identifier.

[0103] Unidentified fragment fingerprints can be generated using the SHA-256 algorithm. If no unidentified fragment exists, a preset null value fingerprint is written. By using a fixed null value fingerprint, it can be ensured that records without unidentified fragments use a consistent null value representation when constructing the third blocking key.

[0104] To generate candidate duplicate record groups, the big data processing platform pre-builds a standard address hierarchy tree. The standard address hierarchy tree includes at least provincial address nodes, city-level address nodes, district / county-level address nodes, street or road nodes, and digital location nodes. Each standard address hierarchy tree node includes at least a node identifier, standard address name, address level, parent node identifier, set of address aliases, and node status.

[0105] In a standard address hierarchy tree, parent-child edges represent hierarchical relationships, and a mutual exclusion node table represents address entries at the same level that cannot simultaneously belong to the same complete address path. For example, the Chaoyang District node and the Haidian District node under the Beijing Municipality node belong to the same administrative level, and they are mutually exclusive within the same complete address path.

[0106] The big data processing platform matches address entries at each level with the standard address hierarchy tree and determines the effective address path with the most consecutive levels and the largest number of covered address segments from the matching results. If multiple candidate paths exist with the same number of covered address segments, each candidate path is saved and further determined based on complete address records and conflicting edges in subsequent duplicate record cluster processing.

[0107] The valid address path for source record R001 is "Beijing / Chaoyang District / Jianguo Road", and the valid address path for source record R007 is "Beijing / Haidian District / Jianguo Road".

[0108] Source record R006 lacks a district-level address entry. Therefore, the missing district-level address level is marked as a pending level, and the corresponding valid address path is saved as "Beijing / Pending District-Level Address Entry / Jianguo Road". Pending levels are not directly considered the same as Chaoyang District or Haidian District, nor are they directly considered mutually exclusive with Chaoyang District or Haidian District. Instead, they are each placed into the corresponding candidate duplicate record group, and the record's ownership is determined jointly by the subsequent complete address path and conflicting edges.

[0109] This embodiment constructs a first blocking key, a second blocking key, and a third blocking key. The first blocking key consists of a core name item and the last-level address item of the valid address path. The second blocking key consists of a core name item, a numeric location item, and the parent address item corresponding to the numeric location item. The third blocking key consists of a name alias item, a valid address path, and an unrecognized fragment content fingerprint.

[0110] Before constructing the blocking key, each component field is converted according to a unified character encoding and connected using a preset delimiter to avoid different field combinations producing the same string. For example, the first blocking key for source record R001 is “**Technology|Jianguo Road”, and the second blocking key is “**Technology|No. 88-Building A-Room 1201|Chaoyang District”. If no name alias field exists, a preset null value is written to the name alias field position in the third blocking key.

[0111] Data records with multiple source names and addresses that share the same first blocking key, the same second blocking key, or the same third blocking key are written to their respective blocking key buckets. A blocking key bucket includes at least a blocking key bucket identifier, a blocking key type, a blocking key string, a set of source record identifiers, and a parent address path identifier.

[0112] The big data processing platform uses each blocking key bucket as a bucket node. When two blocking key buckets contain the same source record identifier, a bucket association edge is established between the two bucket nodes, thus forming a blocking key bucket association graph.

[0113] When determining the closure of a blocking key bucket, the big data processing platform first checks whether the parent address paths corresponding to the bucket nodes are mutually exclusive in the standard address hierarchy tree. Two blocking key buckets with mutually exclusive parent address paths are not merged into the same blocking key bucket closure through bucket association edges; blocking key buckets with no mutual exclusion relationship between their parent address paths and connected through bucket association edges are merged into the same blocking key bucket closure.

[0114] In practical implementation, a disjoint-set data structure can be used to process the association graph of blocking key buckets. Initially, each blocking key bucket forms a separate set. When processing bucket association edges, first check whether the explicit parent address paths in the two sets are mutually exclusive. If there is no mutual exclusion, merge the two sets; if there is mutual exclusion, retain both sets and record the mutual exclusion constraint. After processing, each disjoint-set data structure forms a blocking key bucket closure.

[0115] Source record R006, containing a pending area-level address entry, can appear as a candidate record in both the Chaoyang District blocking key bucket closure and the Haidian District blocking key bucket closure. However, the Chaoyang District blocking key bucket closure and the Haidian District blocking key bucket closure will not be merged based on this. Each blocking key bucket closure forms a candidate duplicate record group. Candidate copies of the same source record identifier in different candidate duplicate record groups still point to the same original multi-source name-address data record.

[0116] The partial blocking bonds formed in this embodiment are shown in Table 4.

[0117] Table 4 Examples of blocking bond construction

[0118] R001 **Technology | Jianguo Road** **Technology | Room 1201, Building A, No. 88 | Chaoyang District Null value | Beijing - Chaoyang District - Jianguo Road | Null value fingerprint R004 **Technology | Jianguo Road** **Technology | Building A, No. 88 | Chaoyang District** **Technology (Beijing) Co., Ltd. | Beijing - Chaoyang District - Jianguo Road | Null Fingerprint R006 **Technology | Jianguo Road** **Technology | Room 1201, Building A, No. 88 | District-level address pending** **Technology | Beijing - District-level Address Item to be Determined - Jianguo Road | Null Fingerprint Value R007 **Technology | Jianguo Road** **Technology | Room 1201, Building A, No. 88 | Haidian District Null value | Beijing - Haidian District - Jianguo Road | Null value fingerprint

[0119] like Figure 5 and Figure 8As shown, the big data processing platform constructs a name-address evidence graph in each candidate duplicate record group. The nodes in the name-address evidence graph correspond to multi-source name-address data records. Consistent edges indicate that there is a positive name-address compatibility relationship between two multi-source name-address data records, while conflicting edges indicate that there is a mutual exclusion relationship between two multi-source name-address data records, meaning they cannot be simultaneously assigned to the same duplicate record cluster.

[0120] A consistent edge is established between two multi-source address data records when the name core entries of two multi-source address data records are the same, or when the name alias entry of one multi-source address data record has a one-to-one alias relationship with the name core entry of another multi-source address data record, and the valid address paths of the two multi-source address data records can be connected to the same candidate parent-child path in the standard address hierarchy tree.

[0121] The one-to-one correspondence between name alias entries and name core entries can come from a pre-established name variant table or from consecutively collected versions of the same source record identifier. The name variant table includes at least a name core entry, a name alias entry, the basis for its establishment, its validity status, and a version identifier.

[0122] For a valid address path containing a pending level, a candidate consistent edge is allowed as long as the pending level can be completed by the complete address paths in the candidate duplicate record group, and there is no mutual exclusion relationship between the currently known level address entry and the numeric location entry. The pending level itself is not used as a basis for two explicit parent address paths to be the same.

[0123] When the core name, alias name, valid address path, and digital location item on which the consistent edge is based are supported by at least two independent evidence domains, or when two multi-source name and address data records belong to consecutive acquisition versions of the same source record identifier and the corresponding digital location items are the same, the corresponding consistent edge will be marked as a reinforced consistent edge.

[0124] When two multi-source address data records have address entries at the same level that correspond to mutually exclusive nodes in the standard address hierarchy tree, a conflict edge is established between the two multi-source address data records; when the same numeric location entries belong to mutually exclusive parent address paths, a conflict edge is also established.

[0125] Both source records R004 and R006 share the core name field "**Technology" and both have the digital location field "No. 88 / Building A / 1201". Furthermore, the pending district-level address field of source record R006 can be completed in the candidate duplicate record group of Chaoyang District. Therefore, a consistent edge is established between source records R004 and R006. A candidate consistent edge can also be formed between source records R006 and R007.

[0126] The district-level address entries of source records R001, R004, and R005 are all in Chaoyang District, and the district-level address entries of source records R007 and R008 are all in Haidian District. Therefore, conflict edges are established between source records R001 and R007, source records R004 and R007, source records R005 and R007, and the corresponding Chaoyang District records and source records R008, respectively.

[0127] Each consistent edge and each conflicting edge is associated with an edge evidence set. The edge evidence set includes at least the edge identifier, edge type, first endpoint source record identifier, second endpoint source record identifier, set of independent evidence domain identifiers, blocking key type, set of field transformation chain identifiers, acquisition time, number of common level address entries, number of supported independent evidence domains, and number of times they are jointly covered.

[0128] The number of independent evidence domains supports a corresponding consistency relationship. Multiple multi-source name-address data records within the same independent evidence domain do not duplicate the number of independent evidence domains supported. The number of common hierarchical address entries represents the number of identical hierarchical address entries corresponding to the same hierarchical position when the name compatibility relationship is established in the records of the two endpoints. The most recent acquisition time is the later of the acquisition times of the two endpoint records.

[0129] The properties of the partially consistent edges formed in this embodiment are shown in Table 5.

[0130] Table 5. Side Evidence Set Attribute Table

[0131] E01 R001—R004 yes 2 3 2026 / 6 / 2 10:00 2 E02 R004—R006 yes 2 2 2026 / 6 / 4 12:00 3 E03 R006—R007 no 1 2 2026 / 6 / 5 14:00 3 E04 R007—R008 yes 2 3 2026 / 6 / 6 14:30 1 E05 R001—R005 yes 2 3 2026 / 6 / 3 11:00 1

[0132] Source records R006 and R007 come from data source SF and data source SG, respectively. However, data source SF and data source SG belong to the same independent evidence domain. Therefore, the number of independent evidence domains supporting the consistent edge E03 is 1, which does not meet the determination condition for anchoring a consistent edge.

[0133] After constructing the name and address evidence graph, the big data processing platform extracts initial consistent edge-connected components formed by consistent edges from the graph. These initial consistent edge-connected components can be determined using either depth-first search or breadth-first search. Starting from an unvisited node, all reachable nodes are visited along the consistent edges, and the visited nodes form an initial consistent edge-connected component.

[0134] If there are no conflicting edges within an initially consistent edge-connected component, where both endpoints are located within that initially consistent edge-connected component, then that initially consistent edge-connected component is considered as a cluster of duplicate records to be verified.

[0135] When both endpoints of a conflicting edge are located within the same initially consistent edge connected component, the conflicting edge is identified as the target conflicting edge, and all shortest consistent edge paths connecting the two endpoints of the target conflicting edge are determined.

[0136] For a consistent edge graph without edge weights, this embodiment uses a breadth-first search algorithm to determine the shortest consistent edge path. During calculation, the first endpoint of the target conflict edge is added to the search queue, and the path distance of the first endpoint is set to 0. Each time, a node is taken from the search queue, and its adjacent nodes are visited along the consistent edge connected to that node. For the first visited adjacent node, the path distance of the adjacent node is set to the current node's path distance plus 1, and the current node is recorded as its predecessor node. For adjacent nodes that have already been visited and whose path distance is equal to the current node's path distance plus 1, the current node is appended as another predecessor node.

[0137] When the second endpoint of the target conflict edge is reached for the first time, the corresponding path distance is the shortest topological distance. By backtracking from the second endpoint back to the first endpoint along the set of predecessor nodes, all shortest consistent edge paths can be obtained.

[0138] In implementations that set non-negative path weights for consistent edges, Dijkstra's algorithm can be used to determine the shortest consistent edge path. Edge weights can be set based on the number of independent evidence fields supported, the number of common level address entries, and the integrity of the field transformation chain, but setting edge weights does not change the mutual exclusion relationship of conflicting edge representations.

[0139] In this embodiment, the shortest consistent edge path between source record R004 and source record R007 is "R004—R006—R007", including consistent edges E02 and E03. A shortest consistent edge path between source record R001 and source record R007 is "R001—R004—R006—R007", including consistent edges E01, E02, and E03. A shortest consistent edge path between source record R005 and source record R007 is "R005—R001—R004—R006—R007", including consistent edges E05, E01, E02, and E03.

[0140] The big data processing platform merges all shortest consistent edge paths corresponding to conflicting edges with the same target, and then takes the union of the merged results corresponding to conflicting edges with different targets to form a conflict subgraph. As can be seen from the aforementioned shortest consistent edge paths corresponding to conflicting edges, consistent edge E03 is covered by the shortest consistent edge paths corresponding to 3 conflicting edges. Therefore, the number of times consistent edge E03 is covered is 3.

[0141] The big data processing platform identifies reinforced consistent edges supported by at least two independent evidence domains and sharing the same corresponding digital location item as anchored consistent edges, and sorts the consistent edges in the conflict subgraph. The sorting uses a step-by-step comparison method: first, it compares whether an edge is an anchored consistent edge; non-anchored consistent edges are ranked before anchored consistent edges. If the types are the same, the number of independent evidence domains supported is compared, with the consistent edge having a smaller value ranked first. If the number of independent evidence domains supported is the same, the number of common hierarchical address items is compared, with the consistent edge having a smaller value ranked first. If the number of common hierarchical address items is the same, the most recent acquisition time is compared, with the earlier consistent edge ranked first. If the most recent acquisition times are the same, the number of common coverages is compared, with the consistent edge having a larger number of common coverages ranked first.

[0142] Based on the attributes shown in Table 5, consistent edge E03 is a non-anchored consistent edge. Consistent edge E03 has one independent evidence domain support and three common coverage times. Therefore, consistent edge E03 is ranked first in the order of removal. The big data processing platform identifies consistent edge E03 as a consistent edge to be removed and removes it from the name-address evidence graph.

[0143] After removing consistent edge E03, the big data processing platform recalculates the consistent edge connected components and re-determines whether there are any target conflict edges whose endpoints are still located in the same consistent edge connected component. This process of removing consistent edges and recalculating connected components is repeated until the two endpoints of each target conflict edge are located in different consistent edge connected components.

[0144] like Figure 8 As shown, before the split, the Chaoyang District record, the pending district-level address record, and the Haidian District record formed an initial consistent edge connected component through consistent edges; after the split, the consistent edge E03 connecting the pending district-level address record and the Haidian District record was removed, and the original initial consistent edge connected component was split into two duplicate record clusters.

[0145] This embodiment ultimately forms a first repeating record cluster and a second repeating record cluster. The first repeating record cluster includes source record R001, source record R002, source record R003, source record R004, source record R005, and source record R006, and the second repeating record cluster includes source record R007 and source record R008.

[0146] After obtaining the duplicate record clusters, the big data processing platform verifies whether there are still conflicting edges within each duplicate record cluster, where both endpoints are located within that cluster. Only when the two endpoints of a conflicting edge are still within the same duplicate record cluster is the corresponding conflicting edge considered an internal conflicting edge. Cross-cluster conflicting edges where the two endpoints are located in different duplicate record clusters are used to maintain the mutual exclusion relationship between the two duplicate record clusters; the corresponding endpoint records are not removed from the duplicate record cluster based on this.

[0147] The big data processing platform identifies multi-source address data records that participate in at least one anchored consistent edge as anchor records. For each multi-source address data record that is not an anchor record, it verifies whether there is a connection path consisting of consistent edges between the multi-source address data record and at least one anchor record, and verifies whether each consistent edge in the connection path has an edge evidence set that can be traced back to the original field value along the field transformation chain.

[0148] Although source records R002 and R003 cannot increase the number of independent supports for the first independent evidence domain, they can be connected to anchor record R001 through consistent edges, and the corresponding consistent edges have complete field transformation chains and source record identifiers. Therefore, source records R002 and R003 are retained in the first duplicate record cluster.

[0149] When a multi-source name-address data record is still an endpoint of an internal conflict edge, cannot be connected to any anchor record through a consistent edge path, or has a consistent edge in the corresponding connection path that cannot be traced back to the original field value, the multi-source name-address data record is removed from the corresponding duplicate record cluster, and the source record identifier, the failed verification type, and the corresponding edge evidence set are written into the abnormal record set.

[0150] After the abnormal records are removed, the big data processing platform aggregates candidate field values ​​according to field type within each duplicate record cluster, and counts multiple supporting records of the same independent evidence domain for the same candidate field value as one independent support.

[0151] The candidate field value table should include at least the candidate field value identifier, the duplicate record cluster identifier, the field type, the candidate field value, the set of supporting source record identifiers, the set of supporting independent evidence fields, the number of consecutive address levels, and the most recent acquisition time.

[0152] Source records R001, R002, and R003 all support the candidate address field value "No. **, Building / Room / **, Jianguo Road, Chaoyang District, Beijing". However, all three records belong to the first independent evidence domain, thus forming only one independent support. Source record R004 belongs to the second independent evidence domain, and source record R005 belongs to the third independent evidence domain, each forming one independent support.

[0153] The big data processing platform constructs candidate standard address paths using only hierarchical address entries and digital location entries that do not have exclusion edges within the same duplicate record cluster, and verifies the parent-child relationship of each candidate standard address path in the standard address hierarchy tree.

[0154] The candidate standard address paths are first sorted in descending order of the number of consecutive address levels. When the number of consecutive address levels is the same, they are then sorted in descending order of the number of independent supports. When both the number of consecutive address levels and the number of independent supports are the same, the current field status is determined according to the most recent collection time in the support record.

[0155] In this embodiment, "No. 88 Jianguo Road, Chaoyang District, Beijing, Building A, Room 1201" has a complete parent-child hierarchical relationship and is supported by the first independent evidence domain, the second independent evidence domain, and the third independent evidence domain. Therefore, this address path is determined as the target standard address path of the first duplicate record cluster.

[0156] The candidate standard address paths are determined as shown in Table 6.

[0157] Table 6. Determination of Candidate Standard Address Paths

[0158] Room 1201, Building A, No. 88 Jianguo Road, Chaoyang District, Beijing 6 3 First, second, and third independent domains of evidence no Selected Room 1201, Building A, No. 88 Jianguo Road, Chaoyang District, Beijing 6 1 Second Independent Evidence Domain no Retain as candidate value Beijing / District-level address to be determined / No. 88 Jianguo Road / Building A / Room 1201 5 1 Fourth Independent Evidence Domain no Participate in parent address determination Room 1201, Building A, No. 88 Jianguo Road, Haidian District, Beijing 6 0 The corresponding record has entered the second duplicate record cluster. yes Not participating in the first duplicate record cluster selection

[0159] For the missing district-level address entry in source record R006, the lower-level address entry "Jianguo Road" corresponding to the missing parent address entry has multiple district-level parent nodes in the standard address hierarchy tree. Therefore, it is not directly supplemented with Chaoyang District or Haidian District. The big data processing platform generates candidate address paths based on each candidate parent address entry and feeds back the mutual exclusion relationships at the same level between different candidate parent address paths to the name-address evidence map.

[0160] When a lower-level address item corresponds to only one parent address item in the standard address hierarchy tree, the unique parent address item is added to the target standard address path, and a completion basis edge is established between the candidate field value node corresponding to the parent address item and the candidate field value node corresponding to the lower-level address item.

[0161] For example, if a standard road node belongs to only one district-level node, and the input address lacks that district-level node, a district-level address entry can be added based on the unique parent-child relationship in the standard address hierarchy tree. During the addition process, the standard address hierarchy tree version, the identifier of the lower-level address entry node, the identifier of the parent address entry node, and the edge used for completion are stored in the field evidence chain.

[0162] When a lower-level address entry corresponds to at least two parent-level address entries, the lower-level address entry is marked as a parent-level ambiguous address entry. Candidate address relationships are established for different candidate parent-level address paths. Conflicting edges are established between mutually exclusive candidate parent-level address paths. Consistent edge connectivity component splitting is re-executed for duplicate record clusters containing parent-level ambiguous address entries.

[0163] The second duplicate record cluster determines the standard address field value "No. **, Building / *, Room / *, Jianguo Road, Haidian District, Beijing" in the same way.

[0164] When determining the standard name field value, the big data processing platform extracts the core name item, name qualification item, and name alias item from multi-source name and address data records that correspond to the same target standard address path and have no internal conflict edges.

[0165] The core name field in the first duplicate record cluster is "**Technology", and the name field values ​​include "Beijing **Technology Co., Ltd.", "**Technology (Beijing) Co., Ltd.", and "**Technology". Among them, "Beijing **Technology Co., Ltd." is supported by the first independent evidence field and the third independent evidence field, "**Technology (Beijing) Co., Ltd." is supported by the second independent evidence field, and "**Technology" is supported by the fourth independent evidence field.

[0166] The big data processing platform determines the core standard name item in descending order of the number of independently supported items. Since the core name item corresponding to all listed name field values ​​is "**Technology", "**Technology" is selected as the core standard name item. Subsequently, administrative region restrictions and organizational form restrictions corresponding to the target standard address path are selected from the name restrictions that support the core standard name item, and combined with the core standard name item in a preset field order to form the standard name field value.

[0167] In this embodiment, the administrative region limitation "Beijing" and the organizational form limitation "Limited Company" are supported by multiple independent evidence fields. Therefore, the standard name core item "**Technology" is combined with the name limitation items "Beijing" and "Limited Company" to form the standard name field value "Beijing **Technology Co., Ltd."

[0168] When a core name item and an alias name item are supported by at least two independent evidence fields, or when the core name item and the alias name item belong to consecutively acquired versions of the same source record identifier and have the same corresponding digital location item, an alias mapping is established between the core name item and the alias name item.

[0169] The complete historical name corresponding to the core item of the standard name, or the complete name in a different order of expression, is written into the alias field of the standard name address record. In this embodiment, "**Technology (Beijing) Co., Ltd." and "**Technology" are written into the alias field of the standard name address record corresponding to the first duplicate record cluster, and the corresponding source record identifier, independent evidence field identifier, and field transformation chain are saved respectively.

[0170] The core field of the standard name in the second duplicate record cluster can also be "**Technology", but the standard name field value and alias field value of the second duplicate record cluster are respectively associated with the target standard address path and the version node of the duplicate record cluster, thus distinguishing it from the first duplicate record cluster.

[0171] When the two endpoints of a newly added conflicting edge contain a standard name core item and a name alias item that has an alias mapping with the standard name core item, respectively, the big data processing platform cancels the corresponding alias mapping, deletes the selection edge corresponding to this alias mapping, and restores the standard name core item, standard name field value, and alias field value before the alias mapping was established along the field evidence chain.

[0172] like Figure 9 As shown, this embodiment uses a directed acyclic structure to store the field evidence chain. The field evidence chain includes source field value nodes, cleansing field value nodes, candidate field value nodes, standard field value nodes, and duplicate record cluster version nodes.

[0173] The source field value node records at least the source record identifier, field type, and original field value; the cleaned field value node records at least the cleaned field value, field transformation chain identifier, and cleansing rule version; the candidate field value node records at least the candidate field value, field type, supported independent evidence domain set, and the identifier of the duplicate record cluster to which it belongs; the standard field value node records at least the standard field value, field type, and validity status; and the duplicate record cluster version node records at least the duplicate record cluster identifier, version number, set of member source record identifiers, and effective time.

[0174] Establish a transformation edge between the source field value node and the cleansing field value node, and record the cleansing operation identifier, cleansing rule version, and execution order. Establish a support edge between the cleansing field value node and the candidate field value node, and record the source record identifier and the independent evidence field identifier. Establish an exclusion edge between the cleansing field value node and the corresponding candidate field value node for data records originating from multi-source name-address data records that have been isolated by internal conflict edges and removed from the corresponding duplicate record cluster. Establish a selection edge between the candidate field value node and the standard field value node. Establish version association edges between the candidate field value node and the standard field value node and the corresponding duplicate record cluster version node, respectively.

[0175] Exclusion edges indicate that the corresponding cleansing field value cannot be supported by the corresponding candidate field value in the current duplicate record cluster version, but do not delete the source field value node and the cleansing field value node. Selection edges connect the selected candidate field value node and the standard field value node in the current duplicate record cluster version.

[0176] Each standard field value node can be traced back to the source field value node corresponding to each independent support along the reverse direction of the selection edge, support edge, and transformation edge, and can determine the corresponding duplicate record cluster version through the version association edge. Candidate field value nodes connected to standard field value nodes through the selection edge do not have a valid exclusion edge pointing to the current duplicate record cluster version.

[0177] The partial field evidence chain of the first duplicate record cluster is shown in Table 7.

[0178] Table 7 Field Evidence Chain Tracing Table

[0179] Standard Name Field Beijing ** Technology Co., Ltd. Beijing ** Technology Co., Ltd. Beijing ** Technology Co., Ltd., Rule V1.3 Beijing ** Technology Co., Ltd. R001 First Independent Evidence Domain Standard Name Field Beijing ** Technology Co., Ltd. Beijing ** Technology Co., Ltd. Beijing ** Technology Co., Ltd., Rule V1.3 Beijing ** Technology Co., Ltd. R005 Third Independent Evidence Domain Standard address field Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing, Rule V1.2 Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing R001 First Independent Evidence Domain Standard address field Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing, Rule V2.1 Room *, Building *, No. ** Jianguo Road, Chaoyang District, Beijing R005 Third Independent Evidence Domain

[0180] Taking the standard address field value corresponding to source record R001 as an example, the complete traceability path of the standard address field value corresponding to source record R001 is as follows: the standard field value node “Beijing Chaoyang District Jianguo Road No.** / * Block / * Room” is connected to the candidate field value node “Beijing Chaoyang District Jianguo Road No.** / * Block / * Room” via a selection edge; the candidate field value node is connected to the cleaned field value node “Beijing Chaoyang District Jianguo Road No.** / * Block / * Room” via a support edge; and the cleaned field value node is connected to the source field value node “Beijing Chaoyang District Jianguo Road No.** / * Block / * Room” via a transformation edge.

[0181] Before standard name and address records are written into the standard name and address database, the big data processing platform performs uniqueness verification, address-level path verification, and field evidence chain tracing verification.

[0182] Uniqueness checks are used to verify whether two standard name-address records with the same unique identifier are generated under the same duplicate record cluster version, and whether different duplicate record clusters generate duplicate unique identifiers due to the same standard name field value and the same complete standard address field value. A unique identifier can be generated by combining the standard name field value, the standard address field value, and the duplicate record cluster identifier.

[0183] Address hierarchy path validation is used to check whether each level of address entry in the standard address field value can be connected continuously in the standard address hierarchy tree according to the parent-child relationship, and to check whether the numeric location entry belongs to the target standard address path.

[0184] The field evidence chain tracing verification is used to check whether each standard field value can be traced back to the corresponding original field value along the selection edge, support edge, and transformation edge, check whether the corresponding candidate field value is associated with the current duplicate record cluster version, and check whether the corresponding candidate field value does not have a valid exclusion edge under the current duplicate record cluster version.

[0185] Standard name and address records that pass all checks are written to the standard name and address database; standard name and address records that fail any check are written to the exception record set, and the type of the failed check, the source record identifier, the candidate field value identifier, and the duplicate record cluster version identifier are saved.

[0186] When receiving newly added, modified, or deleted multi-source address data records, the big data processing platform follows the procedure... Figure 6 The process shown determines the incremental effect on the closure.

[0187] For newly added multi-source name and address data records, perform reversible cleaning and name and address joint feature generation, calculate the first blocking key, second blocking key and third blocking key corresponding to the current record, and determine the corresponding blocking key bucket and the blocking key bucket closure of records that overlap with the corresponding blocking key bucket.

[0188] For the modified multi-source name and address data record, read the blocking key of the record version before the change and the record version after the change respectively, and include the blocking key bucket closure corresponding to the blocking key before the change and the blocking key bucket closure corresponding to the blocking key after the change into the initial influence scope.

[0189] For deleted multi-source name address data records, since the deleted record no longer has a blocking key in the current version, the blocking key, the blocking key bucket to which the record belongs, the associated consistent edge, the associated conflicting edge, and the duplicate record cluster version node of the multi-source name address data record in the previous valid version are read, and the initial scope of influence is determined by the corresponding data in the previous valid version.

[0190] The multi-source address data records in the blocking key bucket closure are identified as initial impact records, and multi-source address data records that can be reached from the initial impact records via consistent edges are added to the impact record set; records connected to multi-source address data records in the impact record set via conflict edges are added to the impact record set; records contained in the shortest consistent edge path connecting the two endpoints of the corresponding conflict edge are added to the impact record set.

[0191] Furthermore, candidate field value nodes and standard field value nodes connected to the version nodes of the duplicate record clusters corresponding to the affected record set through version association edges are added to the affected node set, and the affected record set and the affected node set form an incremental impact closure.

[0192] For example, when the address field of source record R006 changes from "No. **, Jianguo Road, Beijing / Building / Room *" to "No. **, Jianguo Road, Chaoyang District, Beijing / Building / Room *", the big data processing platform determines the blocking key bucket closure of source record R006 before and after the change, and extends the influence on the record set along the consistent edge between source record R006 and source record R004, the original consistent edge between source record R006 and source record R007, the conflicting edge between source record R004 and source record R007, and the corresponding shortest consistent edge path. Candidate field value nodes, standard field value nodes, and duplicate record cluster version nodes related to the first and second duplicate record clusters are all included in the incremental influence closure.

[0193] The big data processing platform only re-executes the candidate duplicate record group generation, name and address evidence graph construction, duplicate record cluster splitting and standard field value determination for the incremental impact closure, while keeping the duplicate record clusters, standard field values ​​and field evidence chains outside the incremental impact closure unchanged.

[0194] For each incremental processing step, the big data processing platform establishes an incremental processing version and records the addition of consistent edges, deletion of consistent edges, addition of conflicting edges, deletion of conflicting edges, duplicate recording of cluster member changes, standard field value changes, and field evidence chain changes.

[0195] The incremental processing version table includes at least the incremental processing version identifier, processing start time, processing end time, affected record set, affected node set, change type, commit status, and the previous incremental processing version identifier. The edge change table records the edge state before and after the change, the duplicate record cluster member change table records the moved-in record and moved-out record, and the standard field value change table records the standard field value before and after the change.

[0196] Before the incremental processing results are written to the standard name and address database, the updated standard name and address records are re-executed for uniqueness verification, address level path verification, and field evidence chain tracing verification.

[0197] When all validations pass, the consistent edges, conflicting edges, duplicate record clusters, standard field values, and field evidence chains corresponding to the incremental processing version are treated as the same transaction and atomically committed. After the commit is complete, the newly added duplicate record cluster version node is marked as the current valid version, and the previous duplicate record cluster version node is marked as a historical version.

[0198] If any verification fails, a rollback is performed based on the incremental processing version records in the reverse order of the change occurrence. During rollback, consistent edges, conflicting edges, selection edges, and duplicate record cluster version nodes newly added in this incremental processing are deleted; consistent edges and conflicting edges deleted in this incremental processing are restored; duplicate record cluster membership relationships in the previous incremental processing version are restored; and standard field values ​​are restored along the field evidence chain corresponding to the previous incremental processing version.

[0199] For example, if the standard address field value generated by incremental processing cannot be traced back to any original address field value along the field evidence chain, the field evidence chain tracing verification fails. The big data processing platform does not submit the result of this incremental processing and restores the previous duplicate record cluster and the corresponding standard address field value according to the incremental processing version record.

[0200] like Figure 10 As shown, the big data processing platform uses multiple data processing nodes to process multi-source name-address data records corresponding to different blocking keys. To ensure that records corresponding to the same blocking key enter the same data processing node, a stable hash value is calculated for the blocking key string, and a modulo mapping is performed according to the number of data processing nodes.

[0201] The target data processing node number is determined according to the following formula:

[0202] (1)

[0203] In the formula, This represents the target data processing node number, which is a dimensionless integer. Represents a stable hash function; Represents the blocking key string; This represents the number of data processing nodes participating in the current task, and is a positive integer.

[0204] Stable hash functions can employ either the MurmurHash3 algorithm or the SHA-256 algorithm. When using the MurmurHash3 algorithm, the hash result is converted to a non-negative integer before performing the modulo operation; when using the SHA-256 algorithm, a predetermined number of bits is extracted from the hash result, converted to an integer, and then the modulo operation is performed.

[0205] For example, if the number of data processing nodes participating in the current task is 3, and the stable hash value of a certain blocking key string is converted into a non-negative integer and modulo 3 to get 1, then the target data processing node number is determined to be 2 according to formula (1).

[0206] After assigning data processing nodes for each blocking key, the big data processing platform establishes a boundary record index for multi-source name-address data records with at least two blocking keys that have been assigned to different data processing nodes. The boundary record index includes at least the source record identifier, blocking key, blocking key type, data processing node identifier, and task version identifier.

[0207] Before constructing the name-address evidence map, each data processing node exchanges candidate record summaries containing the source record identifier, name core item, valid address path, numeric location item, and independent evidence field identifier based on the boundary record index. Candidate duplicate record groups containing the same source record identifier in different data processing nodes are either merged by the distributed task coordination component or rescheduled to the same data processing node for processing.

[0208] After each data processing node completes the consistent edge connectivity component splitting of the corresponding name and address evidence graph, it then determines the cross-node conflict edges of the duplicate record clusters formed by connecting different data processing nodes based on the boundary record index.

[0209] Standard name-address records and field evidence chains are submitted only when the two endpoints of each cross-node conflict edge are located in different duplicate record clusters and the duplicate record cluster versions corresponding to each data processing node are consistent. When the two endpoints of a cross-node conflict edge still belong to the same logical duplicate record cluster, the candidate duplicate record groups involved are rescheduled to the same data processing node, or the cross-node conflict subgraph splitting is performed by the distributed task coordination component.

[0210] To verify the aforementioned processing method, a test dataset containing 1,000,000 multi-source name-address data records was constructed under the same hardware environment. This test dataset includes 12 data sources, four of which have data provision relationships, interface forwarding relationships, or source record replication relationships. The test dataset contains 120,000 pre-labeled name-address objects, including records with the same name but different addresses, records with missing district-level address entries, records with the same road and house number but different parent address paths, and records where the same initial data was repeatedly published from multiple data sources.

[0211] In the test dataset, records with duplication dependencies accounted for 18% of all multi-source address data records, records missing at least one address level accounted for 12%, records with the same name but different addresses accounted for 9%, and records with the same road name and numerical location items but different parent address paths accounted for 6%. The pre-labeled results were generated from the standard address data table and were used to verify the duplicate record clusters and standard field values ​​output by each processing method.

[0212] The test environment consisted of three data processing nodes, each configured with a 16-core processor and 64 GB of RAM. Different processing methods were executed five times each under the same input data and task parameters. The processing time and resource consumption in Tables 8 and 9 are the arithmetic mean of the five execution results.

[0213] The percentage of error standard field values ​​is calculated using the following formula:

[0214] (2)

[0215] In the formula, This indicates the percentage of error standard field values. This indicates the number of standard name and address records whose standard name or standard address field values ​​do not match the pre-annotated results, expressed in records. This indicates the total number of standard name and address records participating in the verification, expressed in records.

[0216] The error merging ratio for conflict logs is calculated using the following formula:

[0217] (3)

[0218] In the formula, This indicates the percentage of conflict record merging errors. This indicates the number of record pairs that have pre-marked conflict relationships but are written to the same duplicate record cluster, expressed in pairs. This represents the total number of pre-labeled conflicting record pairs in the test dataset, expressed in pairs.

[0219] The processing results of different processing methods are shown in Table 8.

[0220] Table 8 Comparison of processing results for different processing methods

[0221] Transitive clustering based on name and address similarity 8.70% 7.90% 451 s Perform a majority determination based on the number of data sources. 6.10% 5.40% 438 s The processing method in this embodiment 1.40% 1.10% 482 s

[0222] Under the aforementioned conditions of constructing test data and hardware, when performing ordinary majority determination based on the number of data sources, the same initial source will generate multiple supports after being forwarded by multiple data sources. However, when using a single count with independent evidence domains, multiple supporting records within the same independent evidence domain will only generate one independent support.

[0223] When using a transitive clustering method based on name and address similarity, records lacking district-level address entries may simultaneously connect to records in Chaoyang District and Haidian District. After removing conflicting edges and the shortest consistent edge path, consistent edges with fewer independent evidence domains and more common coverage are prioritized for removal, allowing records in Chaoyang District and Haidian District to form different clusters of duplicate records.

[0224] Further update processing was performed on 50,000 newly added, modified, or deleted records. The results of full update and incremental update affecting closure update are shown in Table 9.

[0225] Table 9 Performance Comparison of Full Update and Incremental Impact on Closure Update

[0226] Full update 482 s 39.6 GB 18.4 GB Incremental impact on closure update 74 s 8.2 GB 3.1 GB

[0227] The proportion of CPU processing time for incremental updates to CPU processing time for full updates is calculated using the following formula:

[0228] (4)

[0229] In the formula, This represents the percentage of CPU processing time that the increment affects the closure update. This represents the CPU processing time in seconds for the incremental change to affect the closure update. This indicates the CPU processing time for a full update, in seconds.

[0230] Under the test data and hardware conditions constructed in this embodiment, the single count of independent evidence domains reduces the duplicate contribution of data sources with replication dependencies to the number of candidate field value supports; conflict constraint splitting reduces the proportion of records with mutually exclusive parent address paths being grouped into the same duplicate record cluster; incremental influence closure limits the reprocessing scope to the data range that has a blocking key overlap relationship, a consistent edge reachability relationship, a conflict edge relationship, a shortest consistent edge path relationship, or a field evidence chain version association relationship with the changed record.

[0231] In other implementations, multi-source address data records can originate from government databases, business directory databases, map point-of-interest databases, logistics address databases, or customer master data databases. The unified name field can correspond to a business name, organization name, location name, or contact person name, while the unified address field can correspond to a registered address, business address, service address, or delivery address.

[0232] The standard address hierarchy tree can be pre-built based on administrative division data, road data, and house number data, or it can be updated based on an audited standard address database. When updating the standard address hierarchy tree, the standard address hierarchy tree version is set, and the corresponding version is written into the field evidence chain.

[0233] The conditions for establishing consistent edges can be adjusted based on the field types of multi-source address data records, while retaining the negative mutual exclusion relationship represented by conflicting edges. Conflicting edges can be established based on mutual exclusion relationships between address entries at the same level, mutual exclusion relationships between parent address paths, subordinate conflict relationships of digital locations, or confirmed entity exclusion relationships.

[0234] For unweighted address evidence graphs, breadth-first search or bidirectional breadth-first search can be used to determine the shortest consistent edge path. For address evidence graphs where consistent edges have non-negative edge weights, Dijkstra's algorithm can be used to determine the shortest consistent edge path. For conflict subgraphs with a large number of consistent edges, the shortest consistent edge path can be calculated in parallel according to the target conflict edge, and then the shortest consistent edge paths can be merged to form a conflict subgraph.

[0235] The field evidence chain can be stored in a graph database, a relational database, or a key-value database with directed acyclic graph indexing capabilities. Regardless of the storage format, each standard field value can be traced back to its corresponding original field value via selection edges, supporting edges, and transformation edges. Excluded candidate field values ​​do not participate in the selection of standard field values ​​in the current duplicate record cluster version.

[0236] It should be noted that the embodiments of the present invention are not intended to limit the present invention in any way. Any person skilled in the art may use the technical content disclosed in this specification to modify or alter equivalent effective embodiments. Any modifications or equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

[0237] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for cleaning, deduplication, and standardization of multi-source name and address big data, executed by a big data processing platform, comprising: Acquire multi-source name and address data records, divide them into independent evidence domains based on data provision relationships, interface forwarding relationships, and source record replication relationships, and save the source record identifier and original field values; The original field values ​​are mapped to a unified name field and a unified address field and reversibly cleaned. The unified name field is split into a core name item, a name qualifying item, and a name alias item. The unified address field is split into a hierarchical address item, a numeric location item, and an unrecognized fragment, generating a joint name-address feature. Construct blocking keys based on the joint features of name and address, group records with shared blocking keys into candidate duplicate record groups, establish consistent edges based on independent evidence domain support and name-address compatibility, and establish conflict edges based on address mutual exclusion or digital location subordinate conflict relationship to form a name-address evidence graph. For conflicting edges within the same consistent edge connected component, determine the shortest consistent edge path, and determine the consistent edges to be removed according to the number of independent evidence domains supported, the number of common level address entries, and the most recent acquisition time. Remove and recalculate the consistent edge connected component until the two ends of the conflicting edge belong to different consistent edge connected components, and determine the consistent edge connected component as a duplicate record cluster. Within a cluster of repeated records, the supporting records of the same independent evidence domain are counted as only one independent support. The standard field value is determined based on the number of independent supports, the continuity of the address hierarchy, and the collection time, and a standard name and address record and field evidence chain are generated. Perform uniqueness, address level path and field evidence chain tracing verification on standard name and address records. Verify that the candidate field value does not originate from the record isolated by the conflict edge. Write the standard name and address records that pass the verification into the standard name and address database, and write the standard name and address records that fail the verification into the abnormal record set.

2. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 1, characterized in that, The determination of the independent evidence domain includes: Using each of the aforementioned data sources as source nodes, and in accordance with the data flow direction, establish data provision edges, interface forwarding edges, and source record replication edges between the upstream source nodes and the downstream source nodes respectively to form a data source dependency graph; The mutually reachable source nodes in the data source dependency graph are grouped into dependency closure groups, and each dependency closure group is shrunk into the corresponding equivalent source node. In the data source dependency graph that has been shrunk, the source node that has no incoming edges or the equivalent source node is determined as the initial source node; Source nodes that can be traced back to the same initial source node along the reverse direction of the data providing edge, the interface forwarding edge, or the source record copying edge are grouped into the same independent evidence domain.

3. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 2, characterized in that, After performing the reversible cleaning and generating the name-address joint feature, when there is no explicit source record replication relationship between the two source nodes, the source record content fingerprint of each of the multi-source name-address data records is generated according to the name core item, the hierarchical address item, the numeric location item and the original field value respectively. When the fingerprint of each source record in the target acquisition batch of the second source node is the same as the fingerprint of a source record in the previous acquisition batch of the first source node, and the value set of the name core item, the value set of the hierarchical address item, and the value set of the number positioning item in the target acquisition batch of the second source node are respectively included in the corresponding value set in the previous acquisition batch of the first source node, a source record copy edge from the first source node to the second source node is established. The data source dependency graph is updated by copying the newly added source record, and the independent evidence domain containing the first source node and the second source node is redefined.

4. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 1, characterized in that, The reversible cleaning includes: For each original field value, record the field value before cleaning, the field value after cleaning, the cleaning operation identifier, and the cleaning rule version in the order of the cleaning operation execution to form a field transformation chain; The cleaning operation identifier includes at least one of the following: character encoding conversion, full-width / half-width conversion, punctuation normalization, null value placeholder removal, number format unification, field splitting, and field rearrangement. Perform reverse restoration verification on the field transformation chain. Use the cleaned field values ​​that can be restored to the corresponding original field values ​​after the reverse restoration verification to generate the name-address joint feature. Mark the cleaned field values ​​that fail the reverse restoration verification as transformation abnormal field values ​​and stop using the transformation abnormal field values ​​to establish consistent edges.

5. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 4, characterized in that, The construction of the blocking bond includes: The hierarchical address entries are matched with a pre-established standard address hierarchy tree, and the effective address path with the most continuous hierarchical levels and the most covered address segments is determined from the matching results. The name core item and the last-level address item of the valid address path are combined to form a first blocking key. The name core item, the numeric location item, and the parent address item corresponding to the numeric location item are combined to form a second blocking key. An unidentified fragment content fingerprint is generated based on the unidentified fragment. The name alias item, the valid address path, and the unidentified fragment content fingerprint are combined to form a third blocking key. Multi-source name-address data records with the same first blocking key, the same second blocking key, or the same third blocking key are written into the corresponding blocking key buckets respectively; Based on the source record identifier, blocking key buckets with overlapping records are merged into a blocking key bucket closure; The two mutually exclusive blocking key buckets corresponding to the parent address paths in the standard address hierarchy tree are respectively retained in different blocking key bucket closures, and each of the blocking key bucket closures is determined as a candidate duplicate record group.

6. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 5, characterized in that, The establishment of the consistent edge and the conflicting edge includes: When two multi-source address data records have the same name core item, or when one multi-source address data record has a name alias item that has a one-to-one alias relationship with the name core item of another multi-source address data record, and the effective address paths corresponding to the two multi-source address data records can be connected to the same parent-child path in the standard address hierarchy tree, a consistent edge is established between the two multi-source address data records. When the core name, alias name, effective address path and digital location item on which the consistency edge is based are supported by multi-source name and address data records in at least two independent evidence domains, or when the two multi-source name and address data records belong to the consecutive acquisition versions of the same source record identifier and the corresponding digital location items are the same, the corresponding consistency edge is marked as a reinforced consistency edge. When the same level address entries of the two multi-source address data records correspond to mutually exclusive nodes in the standard address hierarchy tree, or the same numeric location entries belong to mutually exclusive parent address paths, a conflict edge is established between the two multi-source address data records. For each consistent edge and each conflicting edge, associate the corresponding source record identifier, independent evidence field identifier, blocking key type, field transformation chain, and acquisition time to form an edge evidence set.

7. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 6, characterized in that, Determining the shortest consistent edge path includes: Extract the initial consistent edge connected component formed by the consistent edge from the name and address evidence graph; Identify a target conflict edge whose two endpoints are both located within the same initially consistent edge connected component; Determine all shortest consistent edge paths connecting the two ends of each target conflict edge, merge all shortest consistent edge paths corresponding to the same target conflict edge, and determine the union of the merge results corresponding to each target conflict edge as the conflict subgraph. The number of independent evidence domains supported by each consistent edge in the conflict subgraph, the number of common level address entries, the most recent acquisition time, and the number of times they are covered by the shortest consistent edge path corresponding to conflicting edges of different targets are statistically analyzed.

8. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 7, characterized in that, Removing the consistent edge to be removed includes: Among the enhanced consistency edges, those supported by at least two independent evidence domains and whose corresponding digital positioning items are the same are identified as anchored consistency edges. The consistent edges in the conflict subgraph are sorted according to whether they are anchored consistent edges, the number of independent evidence domains supported, the number of common level address entries, the most recent acquisition time, and the number of times they are covered together. Non-anchored consistent edges are ranked before anchored consistent edges. The number of independent evidence domains supported and the number of common level address entries are sorted in ascending order, the most recent acquisition time is sorted in ascending order, and the number of times they are covered together is sorted in descending order. The first consistent edge in the sort is identified as the consistent edge to be removed. After removing the consistent edge to be removed, the consistent edge connected component is recalculated, and the target conflict edges whose two endpoints are still located in the same consistent edge connected component are re-identified until the two endpoints of each target conflict edge are located in different consistent edge connected components.

9. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 8, characterized in that, After obtaining the duplicate record cluster, the method further includes: Verify whether there is a conflicting edge within each duplicate record cluster, where both endpoints are located within the duplicate record cluster, and determine the existing conflicting edge as an internal conflicting edge. The multi-source name and address data record that participates in at least one anchored consistent edge is identified as the anchor record; For each multi-source name address data record that does not belong to the anchor record, verify whether there is a connection path consisting of consistent edges between the multi-source name address data record and at least one anchor record, and verify whether each consistent edge in the connection path has an edge evidence set that can be traced back to the original field value along the corresponding field transformation chain. The multi-source name and address data records that serve as endpoints of the internal conflict edge, the multi-source name and address data records that fail the connection path verification, and the multi-source name and address data records that fail the edge evidence set verification are removed from the corresponding duplicate record clusters and written into the abnormal record set. The duplicate record clusters after the records have been removed are used to determine the standard field value.

10. The method for cleaning, deduplication, and standardization of multi-source name and address big data according to claim 9, characterized in that, The field evidence chain adopts a directed acyclic structure and includes source field value nodes, cleansing field value nodes, candidate field value nodes, standard field value nodes, and duplicate record cluster version nodes. A transformation edge is established between the source field value node and the clean field value node; a support edge is established between the clean field value node and the candidate field value node; an exclusion edge is established between the clean field value node and the corresponding candidate field value node, which are derived from multi-source name address data records isolated and removed from the corresponding duplicate record cluster by the internal conflict edge; a selection edge is established between the candidate field value node and the standard field value node; and version association edges are established between the candidate field value node and the standard field value node and the corresponding duplicate record cluster version node, respectively. Each standard field value node can be traced back to the source field value node corresponding to each independent support along the reverse direction of the selection edge, the support edge, and the transformation edge, and the candidate field value node connected to the standard field value node through the selection edge is not connected to the corresponding clean field value node through the exclusion edge.

Images