Place name and address cleaning and matching method and system based on confidence calculation

Abstract

The invention discloses a geographical name address cleaning and matching method and system based on confidence calculation. The system specifically comprises a to-be-cleaned data input interface, a to-be-cleaned data preprocessing module, a hierarchical standard address library, a word segmentation dictionary library, a high-frequency word management module, a cleaning engine containing secondary matching and a confidence calculation module. According to the method, an intelligent place name and address cleaning and matching system including secondary matching, standard address library hot updating and high-frequency word intervention is constructed, the static defect of a traditional hierarchical model in address cleaning and matching is overcome, and precision improvement, efficiency breakthrough and resource optimization are achieved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and more specifically, relates to a place name and address cleaning and matching method and system based on confidence calculation. Background Art

[0002] Place names and addresses are structured or unstructured text that describes the spatial location and attribute information of geographic entities. They typically include key elements such as administrative district name (province / city / district / county), road name, house number, courtyard unit, building number, and geographic coordinates. Their core function is to convert spatial locations described in natural language into standardized, parsable address information, providing fundamental support for geospatial information services.

[0003] As my country accelerates its digitalization and intelligentization efforts, place name and address data has become crucial infrastructure for scenarios such as smart city management, emergency response command, logistics route optimization, and "one-stop" government services. For example, in smart city development, the consistency and currency of place names and addresses are key to information sharing and interaction between different systems and departments. In government emergency management, the accuracy of address information directly impacts the efficiency of emergency response and the rationality of material coordination. In logistics and distribution, the accuracy of address resolution directly determines the timeliness of final delivery and the calculation of logistics costs.

[0004] However, the practical application of place name and address data faces numerous challenges. First, the collection channels for place name and address data include structured databases of government departments (such as the public security standard address database), enterprise-built address databases (which may contain non-standard addresses such as points of interest and aliases), and internet crowd-sourced information (including colloquial descriptions). Data from different sources exhibit significant differences in naming standardization (such as the phenomenon of "Renmin Road" and "Renmin Street" being synonymous with the same object), granularity consistency (missing address elements), and spatiotemporal dynamics (place name changes due to urban renewal). Second, the emergence of new place names and addresses during emergencies requires immediate updates to the standard address database. However, traditional data storage methods are subject to lags and cannot meet the business needs of high dynamics, multi-source integration, and real-time response.

[0005] The current status of relevant technologies for place name and address cleaning and matching is as follows:

[0006] 1) Static hierarchical model. Address elements are parsed step by step through a preset geographic entity hierarchy (such as country / province / city / district / street, etc.). With the help of preset dictionaries of administrative divisions, courtyard units, etc. and regular expression rules, the address text is segmented and matched step by step according to the hierarchical priority, and the hierarchical logical consistency is verified. This type of method performs stably in structured addresses with a high degree of standardization, but when faced with hierarchically missing addresses (missing some address levels, such as the Jinhe Building in Nantong), the matching failure rate increases sharply due to the mechanical reliance on a fixed order. In addition, this technology mainly adopts a single-matching paradigm, that is, the parsing results are directly output through a single round of rule matching. No secondary matching verification mechanism is designed, making it difficult to match fine-grained addresses and unable to achieve accurate spatial positioning.

[0007] 2) Hierarchical correction technology enhanced by statistical models. This technology incorporates statistical learning models (such as inverse document frequency, word vector similarity, and conditional random fields) within the hierarchical matching framework, optimizing matching results based on probability weights. For example, a statistical model is used to sequence address text, identifying hierarchical labels such as "province / city / district / county," and then validating the results using a pre-set hierarchical tree. However, this method relies on the static characteristics of a standard address database. When place-name address data is updated, address resolution will fail because the new address is not promptly incorporated into the standard address database.

[0008] 3) Introducing matching optimization technology for high-frequency words. By introducing high-frequency words, the recognition rate of corresponding address elements can be enhanced. For example, Renmin Road exists in both Chongchuan District and Tongzhou District. By introducing the high-frequency word "Chongchuan District Renmin Road," "Chongchuan District Renmin Road" will be matched first, and "Tongzhou District Renmin Road" will not be matched, thus reducing address ambiguity and improving matching accuracy. Existing technologies limit the scope of high-frequency words to a single level (for example, high-frequency words are only associated with the road level and not the corresponding street information). Cross-level associations are significantly broken. When multiple high-frequency words exist across levels, the inability to link with the associated levels will lead to mismatches of place name and address data.

[0009] The problems currently existing in the prior art are as follows:

[0010] 1) Low matching success rate: Relying solely on single-round parsing results, the remaining text information is not fully utilized, resulting in a low success rate for fine-grained address matching and insufficient spatial positioning accuracy.

[0011] 2) Rigid matching pattern: The matching algorithm relies on a fixed address hierarchy, and the jump address resolution effect is poor for missing layers.

[0012] 3) Imperfect data update mechanism: When the standard address library is updated, the cleaning and matching service needs to be stopped, which has poor flexibility.

[0013] 4) Single confidence assessment factor: Match confidence calculation relies on fixed hierarchical weights and does not consider the impact of spatiotemporal factors such as urban-rural differences, administrative division mergers, and historical place names on confidence;

[0014] 5) Insufficient semantic relevance of high-frequency words: High-frequency words only work at a single address level, lacking cross-level semantic linkage, which easily leads to ambiguity. Summary of the Invention

[0015] Purpose of the invention: The purpose of the present invention is to address the deficiencies in the prior art and to provide a place name and address cleaning and matching method and system based on confidence calculation. This system and method returns standardized and spatialized address information to the user when the place name and address text to be cleaned is input. Through secondary matching, hot update of the standard address database, and intervention of high-frequency words, efficient and accurate place name and address cleaning and matching services are provided.

[0016] Technical solution: The place name and address cleaning and matching method based on confidence calculation described in the present invention includes the following steps:

[0017] S1. Input of data to be cleaned: Build a data input interface. After receiving user request information, parse the input address file to obtain the address text to be cleaned and record the input metadata information.

[0018] S2. Preprocessing of data to be cleaned: preprocessing the input data to be cleaned;

[0019] S3. Hierarchical standard address database: Based on the standard address database of authoritative departments, it integrates multi-source information from alias / historical place name databases and expresses the logical relationship between address elements through a multi-level model;

[0020] S4. Hot update of the word segmentation dictionary: This step is triggered after the hierarchical standard address database is hot updated in step S3. New address entries are acquired and segmented using the Chinese word segmentation engine according to the current dictionary. The segmentation results are pushed to the database administrator within the system. Text information with questionable segmentation is automatically identified. New addresses are marked and added to the dictionary database for hot update.

[0021] S5. High-frequency word management: When a user enters an ambiguous address, the cleaning engine prioritizes high-frequency results based on the corresponding high-frequency word information.

[0022] S6, initial matching: performing initial matching on the data to be cleaned after pre-processing in step S2;

[0023] S7, initial matching result evaluation: Evaluate the initial matching result of step S6 and decide whether to enter the secondary matching process;

[0024] S8, secondary matching: matching the remaining unmatched address texts after the initial matching again;

[0025] S9, confidence calculation: After the matching process is completed, the matching confidence is calculated to reflect the reliability of the matching results;

[0026] S10. Standardized address output: output standardized addresses that conform to a multi-level structure.

[0027] In some embodiments, step S2 specifically includes: first converting the address text into a string that meets the preset hierarchical requirements and is easy to segment based on a regular expression, then using a text deduplication algorithm to delete redundant character information, and using a regularized description mapping table to achieve semantic normalization.

[0028] In some embodiments, step S3 specifically includes: expressing the logical relationship between address elements through a seven-level model to meet the differentiated expression of addresses in urban and rural areas, organizing address entries with a tree-like hierarchical structure, and following the update strategy of regular periodic updates + emergency event triggering, supporting hot updates of the hierarchical standard address library without stopping the cleaning service, and triggering hot updates of the word segmentation dictionary library at the same time.

[0029] In some embodiments, step S5 high-frequency word management specifically includes a high-frequency word database and a high-frequency word management tool. The high-frequency word database is automatically generated based on the high-frequency matching standard addresses in the user's recent cleaning matching results, and is regularly updated with the hierarchical standard address library, including high-frequency word text, corresponding address level, high-frequency word coefficient, timestamp, and activation information.

[0030] In some embodiments, the confidence calculation in step S9 specifically includes: calculating the matching confidence based on the temporal and spatial differences, and the calculation method is: confidence = level coefficient × historical address coefficient × high-frequency word coefficient.

[0031] On the other hand, the present invention also discloses a place name and address cleaning and matching system based on confidence calculation, comprising:

[0032] Input interface for data to be cleaned: Build a data input interface that supports text, API calls, and file input, with built-in metadata information annotation function;

[0033] Data preprocessing module for cleaning: performs preprocessing operations on the input data to be cleaned;

[0034] Hierarchical standard address database: Based on the standard address database of authoritative departments, it integrates multi-source information from alias / historical place name databases and expresses the logical relationship between address elements through a multi-level model;

[0035] Word segmentation dictionary: used to collect, define, and accurately mark the various levels of information of place names and addresses, assisting the Chinese word segmentation engine to correctly segment the address text to be cleaned into hierarchical levels, providing accurate address input for the cleaning engine;

[0036] High-frequency word management module: This module uses a linear transformation function to map high-frequency word coefficients to pre-set confidence intervals, and updates them synchronously with changes in the hierarchical standard address library. It supports manual addition, deletion, modification, and query processing of high-frequency words, realizing the full life cycle management of high-frequency words.

[0037] Cleaning engine with secondary matching: This engine matches the hierarchical segmented text output by the segmentation dictionary library with the hierarchical standard address library step by step. Combined with the high-frequency word coefficient provided by the high-frequency word management module, it increases the probability of preferential matching of addresses that attract high user attention.

[0038] Confidence calculation module: responsible for secondary matching judgment and output of optimal matching results. This module determines whether to trigger secondary matching after evaluating the initial matching results.

[0039] Standardized address output interface: outputs standardized addresses that conform to a multi-level structure.

[0040] In some embodiments, the data preprocessing module specifically includes: performing preprocessing operations including regularization and text deduplication on the data to be cleaned input into the data input interface, performing preliminary correction on common non-standard address information, and removing some data noise.

[0041] In some embodiments, the hierarchical standard address library expresses the logical relationship between address elements through a seven-level model to meet the differentiated expression of addresses in urban and rural areas, organizes address entries in a tree-like hierarchical structure, and follows an update strategy of regular periodic updates + emergency event triggering. It supports hot updates of the hierarchical standard address library without stopping the cleaning service, and at the same time triggers hot updates of the word segmentation dictionary library.

[0042] In some embodiments, the high-frequency word management module specifically includes a high-frequency word database and a high-frequency word management tool. The high-frequency word entries and high-frequency word coefficients in the high-frequency word database are automatically summarized and generated based on the matching frequency of the standard address in the user's recent cleaning matching results. A linear transformation function is used to map the high-frequency word coefficient to a preset confidence interval [0.6, 0.9], and it is updated synchronously with changes in the standard address library.

[0043] In some embodiments, the confidence calculation module specifically includes: calculating the matching confidence according to the temporal and spatial differences, and the calculation method is: confidence = level coefficient × historical address coefficient × high-frequency word coefficient.

[0044] Beneficial effects: The present invention constructs an intelligent place name and address cleaning and matching system that includes secondary matching, hot updating of the standard address database, and the intervention of high-frequency words. It overcomes the static defects of the traditional hierarchical model in address cleaning and matching, and achieves improved accuracy, efficiency breakthroughs, and resource optimization. Compared with the existing technology, the beneficial effects of this application are as follows:

[0045] (1) Improved address matching performance

[0046] Traditional static hierarchical models have a high failure rate in matching hierarchical jump addresses. The cleaning method in this invention no longer simply relies on the fixed hierarchical order of the hierarchical model. It can automatically complete the missing address levels based on the fine-grained matching results, thereby improving the matching completeness.

[0047] In addition, since fine-grained addresses are often located at the end of Chinese address texts, the cleaning engine with secondary matching proposed in the present invention can deeply mine address text information compared to traditional single-round matching, has address error correction function, and can improve the spatial positioning ability of place name addresses.

[0048] (2) Standard address library hot update mechanism improves response efficiency

[0049] The hierarchical standard address library hot update mechanism proposed in the present invention solves the shortcoming of the need to suspend services when updating traditional static models. It supports automatic word segmentation and parsing of newly added addresses, and can instantly add and modify standard address entries in emergency scenarios, thereby improving service flexibility while ensuring system stability.

[0050] (3) Optimizing the address matching evaluation system by using a confidence calculation method that takes into account temporal and spatial differences

[0051] The confidence calculation method proposed in this invention comprehensively considers the differences in urban and rural address structures, uses a differentiation coefficient to calculate the confidence of urban and rural address matching, and superimposes the influence of historical place names and high-frequency words. It is more effective than the evaluation system of existing studies that only relies on hierarchical consistency to obtain confidence.

[0052] (4) High-frequency word diffusion mechanism reduces the probability of address ambiguity

[0053] The high-frequency word diffusion mechanism of the present invention can realize the cross-level transmission of the influence of high-frequency words. For example, when a high-frequency word of a certain community is identified, the address weight of the administrative district where it is located will be increased at the same time, which can reduce the ambiguity and spatial positioning errors caused by the same-name addresses. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a specific flow chart of a matching method according to an embodiment of the present invention;

[0055] Figure 2 This is a specific principle diagram of a matching system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0057] In the description of the present invention, it should be noted that the directions or positional relationships indicated by terms such as "center", "up", "down", "left", "right", "inside" and "outside" are the directions or positional relationships shown, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.

[0058] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0059] The present invention will be further described in detail below through specific implementation examples in conjunction with the accompanying drawings.

[0060] Example 1

[0061] like Figure 1 As shown, a place name and address cleaning and matching method based on confidence calculation includes the following steps:

[0062] S1. Data Input for Cleaning: Build a web data input interface that accepts HTTP requests from users, parses the input address text and address list files to obtain the address text to be cleaned, and records metadata such as the input data source and timestamp. Use an in-memory database and message queue to ensure stable system operation under high concurrency and provide standardized data input for subsequent processing.

[0063] S2. Preprocessing of data to be cleaned: The input data to be cleaned is preprocessed. The preprocessing module integrates regular expression filtering, text deduplication, and regularized description mapping processing algorithms. Specifically, the address text is first converted into a string that meets the preset hierarchical requirements and is easy to segment based on a regular expression. The text deduplication algorithm is then used to delete redundant character information. The regularized description mapping table (such as "Building 3" → "Building 3") is used to achieve semantic normalization, which serves as the input data for the subsequent initial match.

[0064] S3. Hierarchical standard address database: includes standard address data, historical place names, alias address data, etc. Among them, based on the standard address data collected from government authorities, the standard address is defined as an address that conforms to a seven-level continuous expression. The seven levels are: province, city, district / county, street / town, road house number, unit / community / village, building number / group number, which can accurately express the addresses in urban and rural areas (the standard address in an urban area is usually structured as "province / city / district / street / road house number / unit community / building number", such as "Jiangsu Province / Nantong City / Chongchuan District / Wenfeng Street / No. 82 Hongqiao Road / Chengnan New Village / Building 100"; the standard address in a rural area is usually structured as "province / city / county / town / village / village group number", such as "Jiangsu Province / Nantong City / Qidong City / Wangbao Town / Zhengping Village / Group 2 No. 11").

[0065] The standard address is segmented using the Chinese word segmentation engine and thematic word segmentation dictionary. Then, spatial coding technology is used to automatically assign a unique code of fixed length to each hierarchical element and attach the corresponding spatial coordinates. A tree-like hierarchical structure is used to organize address entries, forming a standard address library containing codes, standard address text, and spatial locations.

[0066] Furthermore, by mapping address aliases / historical place names to standard addresses (e.g., "Tongda" → "Nantong University"), non-standard addresses such as aliases and former names can be correctly matched. This hierarchical standard address database supports periodic automatic updates or manual additions and deletions of address entries via hot updates, enabling rapid response to new address cleaning needs in urban renewal scenarios without interrupting service.

[0067] S4. Hot update of word segmentation dictionary: After the hierarchical standard address library is hot updated in step S3, this step is triggered to obtain the newly added address entries and use the Chinese word segmentation engine to perform word segmentation according to the current dictionary. The word segmentation results are pushed to the database administrator within the system, and text information with questionable word segmentation is automatically identified. The newly added addresses are marked and added to the word segmentation dictionary through human-computer interaction to achieve hot update.

[0068] For example, the newly added address "Yi'an Garden, No. 19, Hongqiao Road, Xinchengqiao Street, Chongchuan District" was not included in the original dictionary, and the word segmentation result was "Chongchuan District / Xinchengqiao Street / No. 19, Hongqiao Road / Yi'an / Garden". After recognition, the system provided the database administrator with the questionable address "Yi'an / Garden", which was manually integrated into "Yi'an Garden" and classified into the unit / community / village address level for use by the Chinese word segmentation engine.

[0069] S5. High-Frequency Word Management: High-Frequency Word Management includes a high-frequency word database and a high-frequency word management tool. The high-frequency word database is automatically generated based on the high-frequency matching standard addresses in the user's recent cleaning and matching results. It is regularly updated with the standard address database and includes information such as the high-frequency word text, corresponding address level, high-frequency word coefficient, timestamp, and activation status. When a user enters an address with an unclear administrative area, the cleaning engine prioritizes high-frequency results based on the corresponding high-frequency word information.

[0070] For example, when a user enters the ambiguous address "People's Park", it may correspond to People's Parks in different cities. When the high-frequency word coefficient of "People's Park, Chongchuan District, Nantong City" in the high-frequency word library is higher than others, the cleaning engine will be provided with standard address results in administrative districts that users pay more attention to, avoiding matching unpopular places.

[0071] S6. Initial Matching: First, using the Chinese word segmentation engine and word segmentation dictionary, the address to be cleaned, pre-processed in step S2, is accurately segmented hierarchically. For example, after segmenting "Jinhe Building, No. 78 Chengshan Road, Xinchengqiao Street, Chongchuan District, Nantong City, Jiangsu Province," the address becomes "Jinhe Building, No. 78 Chengshan Road, Xinchengqiao Street, Chongchuan District, Jiangsu Province." Each segmented address is then matched against the tree-like hierarchy in the standard address library, from coarse to fine granularity. When the segmented address is completely consistent with the standard address, the match is considered successful, and all parent address information in the current address hierarchy is automatically completed.

[0072] For example, in the case of "Nantong City / No. 78 Chengshan Road / Jinhe Building," the first address word "Nantong City" successfully matches "Nantong City" in the city-level address hierarchy of the standard address database. Subsequent matches are limited to the administrative area of ​​Nantong City, and the coarse-grained provincial address hierarchy of "Jiangsu Province" is automatically completed. The second address word "No. 78 Chengshan Road" is then matched to the corresponding road address within Nantong City. Subsequent matches are limited to No. 78 Chengshan Road, and the corresponding district and county, "Chongchuan District," and the street to which the road address belongs, "Xinchengqiao Subdistrict," are automatically completed. Finally, the third address word "Jinhe Building" is matched with the standard address. After a successful match, the matching results at all levels are concatenated to form the standard address "Jinhe Building, No. 78 Chengshan Road, Xinchengqiao Subdistrict, Chongchuan District, Nantong City, Jiangsu Province."

[0073] When the high-frequency word coefficient from step S5 is received during matching, for example, if the user inputs "Chengnan New Village", the high-frequency word coefficient of "Chengnan New Village, Chongchuan District" in the high-frequency word library is 0.8, and the high-frequency word coefficient of "Chengnan New Village, Hai'an City" is 0.6. This proves that according to the matching records of the cleaning engine, the user is more inclined to match and will give priority to matching to "Chengnan New Village, Chongchuan District" with a higher coefficient. At the same time, the high-frequency word coefficient of the address within "Chongchuan District" in the user's current matching round will be temporarily increased. When the administrative district of other addresses input by the user is unclear, it will also give priority to matching to the standard address within the administrative district of "Chongchuan District", realizing cross-level transmission of the influence of high-frequency words. The high-frequency word coefficient will also be output to step S9 to calculate the matching confidence.

[0074] After the initial matching is completed, the matching result is output to step S9 for confidence calculation to determine whether a secondary matching is required.

[0075] S7. Evaluation of initial matching results: To improve the efficiency of address cleaning and matching, it is necessary to evaluate the initial matching results and decide whether to enter the secondary matching process. The rules are as follows:

[0076] When all address levels are matched successfully and there are no unmatched address texts, the match is considered complete, the level coefficient is set to 1, and the address is output to step S9 without performing a secondary match. When the address is matched to the building / village group number level, no secondary match is performed regardless of whether there are unmatched address texts. For example, if the input is "Chongchuan District / Chengnan New Village / Building 180 / Room 202", the standard address "Building 180, Chengnan New Village, No. 82 Hongqiao Road, Wenfeng Street, Chongchuan District, Nantong City, Jiangsu Province" is matched, the match has reached the building level, meeting the spatial positioning requirements. Although there is an unmatched address text "Room 202", the level coefficient is set to 1, and the address is output to step S9 without performing a secondary match.

[0077] When the initial match fails to match the building / village group number level and there is unmatched address text, the matched address level is retained and the level parameters are provided to step S9. For example, when an address with an incorrect road number is input as "Nantong City / No. 87 Chengshan Road / Jinhe Building", since "Jinhe Building" is not found under the "No. 87 Chengshan Road" branch (actually located at "No. 78 Chengshan Road"), the matching is terminated and the initial matching result "No. 87 Chengshan Road, Xinchengqiao Street, Chongchuan District, Nantong City, Jiangsu Province" is obtained. Since it is only matched to the road number level and there is unmatched address text "Jinhe Building", Xu outputs the remaining text to step S9 for secondary matching.

[0078] S8. Secondary matching: Fine-grained addresses in Chinese addresses are often located in the second half of the address text. The secondary matching function can fully tap into the remaining unmatched address information. Taking the address with an incorrect road number "Nantong City / No. 87 Chengshan Road / Jinhe Building" as an example, the initial matching result is "No. 87 Chengshan Road, Xinchengqiao Street, Chongchuan District, Nantong City, Jiangsu Province", and the remaining unmatched information is "Jinhe Building". The secondary matching directly inputs the unmatched address for cleaning, and can match it to the standard address with the correct road number "Jinhe Building, No. 78 Chengshan Road, Xinchengqiao Street, Chongchuan District, Nantong City, Jiangsu Province", which can realize the automatic correction of some incorrect addresses. After the secondary matching is completed, the hierarchy coefficient is output to step S9 for comparison with the hierarchy coefficient of the initial matching result.

[0079] In addition, since individual building numbers / village group numbers (such as "Building 25" and "No. 14, Group 5") are usually not unique, their upper-level units / communities / villages cannot be directly inferred. For example, if an address with an incorrect building number is entered as "Building 300, Chengnan New Village, Chongchuan District" (this building number does not exist in this community), the initial matching result is "Chengnan New Village, No. 82, Hongqiao Road, Wenfeng Street, Chongchuan District, Nantong City, Jiangsu Province", and the unmatched text used in the second match is "Building 300". The matching engine will control the building number level to not skip the unit / community and directly match the text to avoid mistaken matching to other communities.

[0080] S9. Confidence Calculation: After the matching process is completed, the matching confidence is calculated to reflect the reliability of the matching results. The matching engine comprehensively considers the differences in standard address structures between urban and rural areas, the impact of historical place names on address validity, and the potential address ambiguity caused by addresses with the same name in different locations. The matching confidence is calculated based on temporal and spatial differences. The calculation method is: Confidence = Hierarchy coefficient × Historical address coefficient × High frequency word coefficient. The calculation rules for each coefficient are as follows:

[0081] The hierarchy coefficient is determined by the finest-grained address level of the matching result. Different calculation methods are used for the hierarchy coefficients of urban and rural addresses based on the address spatial positioning capability. When the finest-grained address level of the matching result is the province, city, district / county, or street / town level, since spatial positioning cannot be performed, the match is judged to have failed and the hierarchy coefficient is set to 0; when the finest-grained address level of the matching result is the building / village group number, the address can be accurately located and the hierarchy coefficient is set to 1. In cases where only rough positioning can be achieved, differentiated values ​​are assigned based on the urban and rural attributes of the address. When the finest-grained address level of an urban address is the road number and the unit / community, the hierarchy coefficients are set to 0.7 and 0.8, respectively; when the finest-grained address level of a rural address is the village, the hierarchy coefficient is set to 0.6.

[0082] The default value of the historical address coefficient is 1. When the matching result is a historical place name, the time difference between the extinction time of the historical place name and the user query time is calculated, and the historical address coefficient is mapped to the preset interval [0.6, 1) using a linear function. The lower the value, the less current the matching result is, and the matching confidence is suppressed.

[0083] The high-frequency word coefficient is set to 1 by default. When the high-frequency word coefficient of step S5 is received during the matching process, this value is used to suppress the matching confidence and trigger a system prompt due to the possible ambiguity of "same name in different places". It is recommended that users supplement administrative division information to improve accuracy.

[0084] When the confidence is calculated according to the above formula, if there is only a single matching result, the confidence and matching result are directly output to step S10; if there is a secondary match, the two matching results are compared. If the initial matching and secondary matching results belong to the same district or county-level administrative area, the result with higher confidence is selected and output to step S10. Otherwise, the initial matching result is still output to step S10 to reduce the probability of false matching.

[0085] For example, if the address to be cleaned is "Chengnan New Village, No. 83, Hongqiao Road, Chongchuan District" (wrong house number), the initial matching result is "No. 83, Hongqiao Road, Wenfeng Street, Chongchuan District, Nantong City, Jiangsu Province", and the secondary matching result is "Chengnan New Village, No. 124, Changjiang Middle Road, Hai'an Street, Hai'an City, Nantong City, Jiangsu Province". Although the secondary matching result has a granularity of the unit community level and is better than the initial matching result, the two matches do not belong to the same district or county administrative area. In this case, the initial matching result is still output to avoid misjudgment.

[0086] S10. Standardized address output: Outputs standardized addresses that conform to a seven-level structure. Output in JSON and Excel file formats is supported. The output results include complete address information, hierarchical address information, spatial coordinates, matching confidence, high-frequency word impact parameters, and other information, providing data support for subsequent analysis.

[0087] Example 2

[0088] like Figure 2 As shown, a place name and address cleaning and matching system based on confidence calculation includes:

[0089] ① Input interface for data to be cleaned: Build a web data input interface that supports text, API calls, and file (CSV / XLS) input. It has built-in metadata annotation functions (recording information such as source and system timestamp), providing a standardized data entry for subsequent cleaning processes and directly connecting to the ② pre-processing module for data to be cleaned.

[0090] ② Preprocessing module for data to be cleaned: performs preprocessing operations including regularization and text deduplication on the data to be cleaned input into the ① data to be cleaned input interface, makes preliminary corrections to common non-standard address information, removes some data noise, and outputs the preprocessed addresses to be cleaned to the ④ (address and place name topic) word segmentation dictionary library.

[0091] ③ Hierarchical standard address database: As the data foundation of the place name and address cleaning and matching system, it is based on the standard address database of the authoritative department, integrates multi-source information such as alias database and historical place name data, and expresses the logical relationship between address elements through a seven-level model (the seven levels are: province, city, district / county, street / town, road house number, unit / community / village, building number / village group number). It can meet the differentiated expression of addresses in urban and rural areas, organizes address entries in a tree-like hierarchical structure, and supports hot updates of the hierarchical standard address database without stopping the cleaning service according to the update strategy of regular periodic updates + emergency triggers. At the same time, it triggers the hot update of the ④ word segmentation dictionary library.

[0092] ④ The word segmentation dictionary is a database specifically constructed for place name and address text parsing. It collects, defines, and accurately labels each level of information for place names and addresses. This assists the Chinese word segmentation engine in correctly segmenting the addresses to be cleaned from the ② pre-processing module into hierarchical address text, providing accurate address input for the ⑥ cleaning engine with secondary matching. For example, it can split "Jinhe Building, Chongchuan District, Nantong City" into "Nantong City / Chongchuan District / Jinhe Building." When new address entries are added to the ③ hierarchical standard address database due to a hot update, a hot update of the ④ word segmentation dictionary is automatically triggered. This new dictionary entry is added without restarting the Chinese word segmentation engine, ensuring accurate word segmentation of the addresses to be cleaned.

[0093] ⑤ High-frequency word management module: includes a high-frequency word database and a high-frequency word management tool. The high-frequency word entries and high-frequency word coefficients in the high-frequency word database are automatically summarized and generated based on the matching frequency of the standard address in the user's recent cleaning and matching results. The high-frequency word coefficients are mapped to the preset confidence interval [0.6, 0.9] using a linear transformation function, and are updated synchronously with the changes in the ③ hierarchical standard address library. It supports manual addition, deletion, modification, and query of high-frequency words, realizing the full life cycle management of high-frequency words. When the user-input address does not contain administrative district information and may lead to mismatching due to ambiguity, the high-frequency word coefficients are pushed to the ⑥ cleaning engine with secondary matching.

[0094] ⑥ Cleaning Engine with Secondary Matching: This engine performs a level-by-level match between the hierarchical segmented text output by the ④ Segmentation Dictionary and the ③ Hierarchical Standard Address Library. Combined with the high-frequency word coefficients provided by the ⑤ High-Frequency Word Management Module, this engine prioritizes matching addresses of high user interest. The hierarchical matching results are then sent to the ⑦ Confidence Calculation Module. If the ⑦ Confidence Calculation Module determines that a second match is still necessary, the remaining address text after the initial match is again hierarchically segmented and matched level-by-level against the ③ Hierarchical Standard Address Library. The results of the second match are also sent to the ⑦ Confidence Calculation Module.

[0095] ⑦ Confidence Calculation Module: This module is one of the core modules of the system, primarily responsible for determining secondary matches and outputting the optimal match result. This module evaluates the initial match results and determines whether to trigger a secondary match. If the initial match meets the precise positioning requirements (the finest-grained address level of the matching result reaches the building / village group number) and contains no unmatched text, the confidence level is directly calculated and the result is output to the ⑧ Standardized Address Output Interface. If the match result does not meet the precise positioning requirements and there is unmatched text, the remaining address text is fed back to the ⑥ Cleaning Engine with Secondary Matching for a secondary match. Finally, by comparing the confidence values ​​of the two matches and the administrative districts they are located in, if the two matches are within the same district / county-level administrative district, the result with the higher confidence level is selected and output to the ⑧ Standardized Address Output Interface; otherwise, the initial match result is still output.

[0096] The confidence calculation method is: confidence = hierarchy coefficient × historical address coefficient × high-frequency word coefficient.

[0097] Among them, the hierarchy coefficient is determined by the matching result and the finest-grained address hierarchy. A fixed hierarchy coefficient is assigned to each hierarchy according to the urban and rural category of the address. The rules are as follows: When the finest-grained address hierarchy of the matching result is the province, city, district / county, street / town level, since spatial positioning cannot be performed, it is judged that the match has failed, and the hierarchy coefficient is set to 0; when the finest-grained address hierarchy of the matching result is the building / village group number, the address can be accurately located, and the hierarchy coefficient is set to 1. In the case where only rough positioning can be achieved, differentiated values ​​are assigned according to the urban and rural attributes of the address. When the finest-grained address hierarchy of the urban address is the road number and the unit / community, the hierarchy coefficients are set to 0.7 and 0.8 respectively; when the finest-grained address hierarchy of the rural address is the village, the hierarchy coefficient is set to 0.6.

[0098] The default value of the historical address coefficient is 1. When the matching result is a historical place name, the time difference between the extinction time of the historical place name and the user query time is calculated, and the historical address coefficient is mapped to the preset interval [0.6, 1) using a linear function. The lower the value, the less current the matching result is, and the matching confidence is suppressed.

[0099] The high-frequency word coefficient is set to 1 by default. When the high-frequency word coefficient of the word segmentation dictionary is received during the matching process, this value is used to suppress the matching confidence and trigger a system prompt due to the possible ambiguity of "same name in different places". It is recommended that users supplement administrative division information to improve accuracy.

[0100] ⑧ Standardized address output interface: Based on the results of ⑥ cleaning engine with secondary matching, the final multi-level standardized address is generated, supporting JSON / XLS file output, and additional information such as matching confidence, whether it is a historical place name, whether the administrative division is questionable, and processing timestamp.

[0101] The present invention constructs an intelligent place name and address cleaning and matching system that includes secondary matching, hot updating of the standard address database, and intervention of high-frequency words. It overcomes the static defects of the traditional hierarchical model in address cleaning and matching, and achieves accuracy improvement, efficiency breakthrough and resource optimization.

[0102] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A place name and address cleaning and matching method based on confidence calculation, characterized by: The steps include: S1. Input of data to be cleaned: Build a data input interface. After receiving user request information, parse the input address file to obtain the address text to be cleaned and record the input metadata information. S2. Preprocessing of data to be cleaned: preprocessing the input data to be cleaned; S3. Hierarchical standard address database: Based on the standard address database of authoritative departments, it integrates multi-source information from alias / historical place name databases and expresses the logical relationship between address elements through a multi-level model; S4. Hot update of the word segmentation dictionary: This step is triggered after the hierarchical standard address database is hot updated in step S3. New address entries are acquired and segmented using the Chinese word segmentation engine according to the current dictionary. The segmentation results are pushed to the database administrator within the system. Text information with questionable segmentation is automatically identified. New addresses are marked and added to the dictionary database for hot update. S5. High-frequency word management: When a user enters an ambiguous address, the cleaning engine prioritizes high-frequency results based on the corresponding high-frequency word information. S6, initial matching: performing initial matching on the data to be cleaned after pre-processing in step S2; S7, initial matching result evaluation: Evaluate the initial matching result of step S6 and decide whether to enter the secondary matching process; S8, secondary matching: matching the remaining unmatched address texts after the initial matching again; S9, confidence calculation: After the matching process is completed, the matching confidence is calculated to reflect the reliability of the matching results; S10. Standardized address output: output standardized addresses that conform to a multi-level structure.

2. The place name and address cleaning and matching method based on confidence calculation according to claim 1 is characterized in that: Step S2 specifically includes: first, converting the address text into a character string that meets the preset hierarchical requirements and is easy to segment based on a regular expression, then using a text deduplication algorithm to delete redundant character information, and using a regularized description mapping table to achieve semantic normalization.

3. The place name and address cleaning and matching method based on confidence calculation according to claim 1 is characterized in that: Step S3 specifically includes: expressing the logical relationship between address elements through a seven-level model to meet the differentiated expression of addresses in urban and rural areas, organizing address entries with a tree-like hierarchical structure, and following the update strategy of regular periodic updates + emergency event triggers. It supports hot updates of the hierarchical standard address library without stopping the cleaning service, and triggers hot updates of the word segmentation dictionary library at the same time.

4. The place name and address cleaning and matching method based on confidence calculation according to claim 1 is characterized in that: Step S5 high-frequency word management specifically includes a high-frequency word database and a high-frequency word management tool. The high-frequency word database is automatically generated based on the high-frequency matching standard addresses in the user's recent cleaning matching results, and is regularly updated with the hierarchical standard address library, including high-frequency word text, corresponding address level, high-frequency word coefficient, timestamp, and activation information.

5. The place name and address cleaning and matching method based on confidence calculation according to claim 1 is characterized in that: Step S9 confidence calculation specifically includes: calculating the matching confidence based on the temporal and spatial differences, and the calculation method is: confidence = level coefficient × historical address coefficient × high-frequency word coefficient.

6. A place name and address cleaning and matching system based on confidence calculation, characterized by: include: Input interface for data to be cleaned: Build a data input interface that supports text, API calls, and file input, with built-in metadata information annotation function; Data preprocessing module for cleaning: performs preprocessing operations on the input data to be cleaned; Hierarchical standard address database: Based on the standard address database of authoritative departments, it integrates multi-source information from alias / historical place name databases and expresses the logical relationship between address elements through a multi-level model; Word segmentation dictionary: used to collect, define, and accurately mark the various levels of information of place names and addresses, assisting the Chinese word segmentation engine to correctly segment the address text to be cleaned into hierarchical levels, providing accurate address input for the cleaning engine; High-frequency word management module: This module uses a linear transformation function to map high-frequency word coefficients to pre-set confidence intervals, and updates them synchronously with changes in the hierarchical standard address library. It supports manual addition, deletion, modification, and query processing of high-frequency words, realizing the full life cycle management of high-frequency words. Cleaning engine with secondary matching: This engine matches the hierarchical segmented text output by the segmentation dictionary library with the hierarchical standard address library step by step. Combined with the high-frequency word coefficient provided by the high-frequency word management module, it increases the probability of preferential matching of addresses that attract high user attention. Confidence calculation module: responsible for secondary matching judgment and output of optimal matching results. This module determines whether to trigger secondary matching after evaluating the initial matching results. Standardized address output interface: outputs standardized addresses that conform to a multi-level structure.

7. The place name and address cleaning and matching system based on confidence calculation according to claim 6 is characterized in that: The data preprocessing module specifically includes: performing preprocessing operations including regularization and text deduplication on the data to be cleaned input into the data input interface, performing preliminary correction on common non-standard address information, and removing some data noise.

8. The place name and address cleaning and matching system based on confidence calculation according to claim 6 is characterized in that: The hierarchical standard address database expresses the logical relationship between address elements through a seven-level model to meet the differentiated expression of addresses in urban and rural areas. It adopts a tree-like hierarchical structure to organize address entries and follows a regular periodic update + emergency event triggering update strategy. It supports hot updates of the hierarchical standard address database without stopping the cleaning service, and triggers hot updates of the word segmentation dictionary database at the same time.

9. The place name and address cleaning and matching system based on confidence calculation according to claim 6 is characterized in that: The high-frequency word management module specifically includes a high-frequency word database and a high-frequency word management tool. The high-frequency word entries and high-frequency word coefficients in the high-frequency word database are automatically summarized and generated based on the matching frequency of the standard address in the user's recent cleaning matching results. A linear transformation function is used to map the high-frequency word coefficient to a preset confidence interval [0.6, 0.9], and it is updated synchronously with changes in the standard address library.

10. The place name and address cleaning and matching system based on confidence calculation according to claim 6, characterized in that: The confidence calculation module specifically includes: calculating the matching confidence according to the time-space difference, and the calculation method is: confidence = level coefficient × historical address coefficient × high-frequency word coefficient.

Images