Data cleaning method and system for address information

Abstract

The application relates to the technical field of information processing, and provides a data cleaning method and system for address information. The data cleaning method comprises the following steps: carrying out standardization pretreatment on original address text recognized by OCR, stripping redundant noise, regularizing format, extracting core address information, searching administrative division by using a multilevel fault-tolerant matching mechanism, fusing buffer quick query, accurate string matching and text semantic vector bottom-up matching, and superimposing administrative level correlation verification to eliminate recognition errors; after the administrative division verification is correct, completing national standard administrative code conversion according to a region code table, and outputting standardized structured address data. The application constructs an address cleaning pipeline integrating pretreatment noise reduction, multilevel matching, hierarchical logic verification, international code conversion and structured export, relies on the architecture design of multilevel matching complementation and hierarchical verification constraint, realizes automatic regular processing from non-standard address to national standard code, and improves data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to a data cleaning method and system for address information. Background Technology

[0002] In current vehicle-related operations, the issuing authority information on vehicle registration certificates relies on manual input of provincial and municipal codes, which is inefficient and prone to errors. Although OCR technology has been used to extract text, the recognition results generally suffer from quality defects such as typos, missing characters, and mixed non-administrative region keywords. Furthermore, the industry lacks a complete address cleaning solution for such defective text, resulting in the need for manual verification at the business end, making it impossible to achieve automated and standardized acquisition of provincial and municipal codes. Summary of the Invention

[0003] In view of this, this application aims to propose a data cleaning method for address information, so as to realize the automated normalization process of OCR recognition text with quality defects into standardized encoding, thereby improving processing efficiency and matching accuracy.

[0004] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0005] A data cleaning method oriented towards address information includes:

[0006] The original text obtained by OCR recognition is standardized and preprocessed to obtain the core target text;

[0007] Using the core target text, a hierarchical matching strategy is executed to obtain the corresponding administrative region information;

[0008] Based on the administrative region information obtained through matching, the corresponding national standard code is mapped from the regional code table, and a structured parsing result is output.

[0009] The hierarchical matching strategy adopts a strategy of prioritizing exact matching and using text vector matching as a fallback.

[0010] Furthermore, the standardization preprocessing includes:

[0011] The original text is filtered for valid characters, removing non-Chinese characters and retaining only Chinese characters.

[0012] Based on a pre-set stop word library, the filtered text is subjected to keyword matching and truncation, and the matched stop words and their subsequent characters are deleted to obtain the core target text.

[0013] Furthermore, the stop word library adopts a hot configuration mechanism, including:

[0014] The stop word library is centrally managed by the configuration center and supports adding, modifying or deleting stop words online.

[0015] The configuration center pushes changes to the stop word library to the local service in real time;

[0016] The local service dynamically updates the local cached stop word library after receiving push notifications.

[0017] Furthermore, before executing the hierarchical matching strategy, the data cleaning method also includes performing a cache query, and the cache query includes:

[0018] Using the core target text as the key, query the local cache library, wherein the local cache library adopts a key-value pair storage structure, the key is the core target text, and the value is the encoding mapping result corresponding to the text;

[0019] When a valid cache entry with the core target text as the key exists in the local cache library, it is determined as a cache hit. The encoding mapping result stored in the cache entry is read, and the read encoding mapping result is directly output as the structured parsing result.

[0020] If no cache entry with the core target text as the key exists in the local cache library, it is determined as a cache miss, and the hierarchical matching strategy continues to be executed.

[0021] Furthermore, the hierarchical matching strategy includes:

[0022] Perform exact matching based on left-hand character matching, and perform association validation by level;

[0023] If an exact match fails, determine whether the text vector matching switch is enabled.

[0024] Specifically, when the text vector matching switch is turned on, text vector matching based on cosine similarity is performed, and hierarchical linkage verification is carried out.

[0025] Returns an empty value if the text vector matching switch is off or if vector matching fails.

[0026] Furthermore, the execution of precise matching based on left-hand character matching includes:

[0027] The core target text is left-matched with the provincial administrative region names in the regional code table. After a successful match, the provincial information is extracted, and the matched provincial names in the core target text are deleted to generate the remaining text.

[0028] The remaining text is left-matched with the city-level administrative region names in the regional code table. If the match is successful, the city-level information and the corresponding city-level code are extracted.

[0029] Verify whether the superior code of the city-level code obtained by matching is consistent with the provincial code.

[0030] Furthermore, the text vector matching based on cosine similarity includes:

[0031] The text to be matched is processed by Chinese word segmentation, which divides the text into several word sequences, and the word sequences are converted into fixed-dimensional numerical vectors through a pre-trained vectorization model.

[0032] Calculate the cosine similarity between the numerical vector and the corresponding administrative region name vector in the regional code table;

[0033] The cosine similarity calculation result is compared with the preset cosine similarity configuration threshold. Candidate results with a cosine similarity greater than the preset cosine similarity configuration threshold are selected, and the selected candidate results are subjected to hierarchical linkage verification.

[0034] Furthermore, the cosine similarity threshold supports dynamic configuration and employs different cosine similarity thresholds based on the OCR-recognized text quality score: and / or,

[0035] The cosine similarity threshold can be dynamically configured and adjusted in real time according to the matching accuracy requirements of the current business scenario.

[0036] Furthermore, the regional code table stores the standard names, national standard codes, and hierarchical relationships of provincial and municipal administrative divisions in a structured manner;

[0037] Municipalities and special administrative regions are managed at the provincial level and are not divided into city levels, while autonomous regions are managed at the provincial level and include data on their subordinate city levels.

[0038] Compared with related technologies, this application has the following advantages:

[0039] (1) The address information-oriented data cleaning method of this application sequentially performs standardized preprocessing on the original text obtained by OCR recognition to obtain core target text, uses the core target text to perform a hierarchical matching strategy with precise matching as the priority and text vector matching as the fallback to match the corresponding administrative region information, and maps the corresponding national standard code from the regional code table according to the matched administrative region information and outputs a structured parsing result. By constructing a dedicated automated data cleaning system for OCR-recognized text of vehicle registration certificates with quality defects such as typos, missing characters, and mixed irrelevant keywords, it can realize the full-chain automated processing from non-standardized defective text to standardized provincial and municipal national standard codes, replacing the traditional manual verification and entry mode. The processing time of a single data entry is reduced from minutes to milliseconds, solving the technical problems of low efficiency, high error rate, and high cost of manual operation.

[0040] (2) By effectively filtering non-Chinese characters from the original text and combining the pre-set stop word library for keyword matching and truncation to delete stop words and their subsequent characters, it is possible to eliminate the interference of non-administrative region keywords and irrelevant characters from the source, improve the accuracy of subsequent matching, and lay the foundation for high-quality matching.

[0041] (3) The stop word library is centrally managed through the configuration center and supports online addition, deletion and modification of stop words. Combined with the configuration center, changes are pushed to the local service in real time and the local service dynamically updates the cache. This enables the stop word library to take effect immediately without restarting the system, improving the system's ability to quickly adapt to keyword changes in the text of the issuing authority and ensuring the timeliness and flexibility of the filtering logic.

[0042] (4) By querying the local cache library based on the key-value pair storage structure with the core target text as the key, the encoded mapping result is read directly when the cache is hit and the subsequent matching process is skipped, which can avoid repeated text matching calculations, improve parsing efficiency in high-concurrency business scenarios, and reduce the consumption of system server resources.

[0043] (5) By performing precise matching based on character left matching and performing association verification by level, when precise matching fails, it determines whether the text vector matching switch is turned on. When it is turned on, it performs text vector matching based on cosine similarity and performs hierarchical linkage verification. This can realize the strategy of prioritizing precise matching and using text vector matching as a fallback, and adapt to various quality defect scenarios such as typos and missing characters in OCR recognition text.

[0044] (6) By left-matching the core target text with the provincial administrative region name in the regional code table, and left-matching the remaining text with the municipal administrative region name after successful matching, and performing consistency verification between the municipal code and the provincial code, it is possible to achieve accurate matching of provincial-municipal level and mandatory verification of hierarchical association, thus avoiding the error result of mismatch between province and city.

[0045] (7) By performing Chinese word segmentation on the text to be matched and converting it into a fixed-dimensional numerical vector, the cosine similarity between the vector and the corresponding administrative region name vector in the regional code table is calculated, and candidate results with a value greater than the preset threshold are selected for hierarchical linkage verification. This enables the implementation of non-precise matching as a fallback for scenarios where precise matching fails, thus solving the problem of precise matching failure caused by OCR recognition text misspellings and missing characters.

[0046] (8) By supporting different cosine similarity thresholds for OCR-recognized text quality scores and / or adjusting the thresholds in real time according to the requirements of matching accuracy in the current business scenario, the system can achieve differentiated adaptation of matching accuracy and recall, flexibly adjust the matching strategy according to the quality of the text and business needs, and improve the system's adaptability in different business scenarios.

[0047] (9) By storing the standard names, national standard codes and hierarchical relationships of provincial and municipal administrative divisions in a structured manner, and by managing municipalities and special administrative regions at the provincial level without dividing them into city levels, and managing autonomous regions at the provincial level and including data of their subordinate cities, it is possible to achieve complete coverage and coding standardization of all statutory administrative division scenarios, and provide an accurate and reliable core data source for coding mapping.

[0048] Another objective of this application is to propose a data cleaning system for address information, comprising:

[0049] The preprocessing module is used to standardize and preprocess the raw text obtained by OCR recognition to obtain the core target text;

[0050] The hierarchical matching module is used to execute a hierarchical matching strategy using the core target text to match and obtain the corresponding administrative region information;

[0051] The encoding mapping module is used to map the corresponding national standard code from the regional code table based on the matched administrative region information, and output a structured parsing result;

[0052] The hierarchical matching strategy adopts a strategy of prioritizing exact matching and using text vector matching as a fallback.

[0053] The address information-oriented data cleaning system of this application, through the coordinated operation of a preprocessing module, a hierarchical matching module, and an encoding mapping module, performs standardized preprocessing, hierarchical matching with a fallback strategy of prioritizing precise matching and text vector matching, and encoding mapping and structured output based on regional code tables, respectively, and can realize the fully automated and standardized processing from OCR-recognized text to standardized national standard encoding. Attached Figure Description

[0054] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a schematic diagram of the overall process of the address-oriented data cleaning method described in the embodiments of this application;

[0056] Figure 2 This is a schematic diagram of the structure of the address-oriented data cleaning system described in an embodiment of this application.

[0057] Explanation of reference numerals in the attached figures:

[0058] 410 - Preprocessing module; 420 - Layered matching module; 430 - Encoding mapping module. Detailed Implementation

[0059] To make the technical solution and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0061] Furthermore, it should be noted that in the description of this application, if terms such as "upper," "lower," "inner," or "outer" appear, indicating orientation or positional relationship, these are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, if terms such as "first" or "second" appear, they are also used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0062] Furthermore, in the description of this application, unless otherwise expressly defined, the terms "installation," "connection," "joining," and "connector" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application in light of the specific circumstances.

[0063] In this application, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0064] The present application will now be described in detail through exemplary embodiments. However, it should be understood that, without further description, elements, structures, and features in one embodiment may be advantageously incorporated into other embodiments.

[0065] It should be noted that OCR (Optical Character Recognition) is a technology that automatically converts visual text in an image into editable and searchable text data. Specifically, it refers to the process of converting text in images, scanned documents, or PDFs into computer-editable and searchable text formats. In this application, it specifically refers to extracting the original text of the issuing authority field from a vehicle registration certificate photograph.

[0066] The first aspect of this application provides a data cleaning method for address information, which realizes fully automated processing of OCR-recognized text with quality defects such as typos, missing characters, and irrelevant keywords into standardized national standard codes. This solves the technical problems of low efficiency, high error rate, and high cost of manual operation, as well as the technical gap in the lack of a dedicated data cleaning solution for such defective text in the prior art.

[0067] While OCR technology has been used to automatically extract the issuing authority text from vehicle registration certificates, it only performs a basic image-to-text conversion without any subsequent analysis or encoding mapping of the recognition results. Furthermore, due to factors such as the clarity of the vehicle registration certificate stamp and the shooting angle, the OCR-recognized issuing authority text generally suffers from typos, missing characters, and omissions. Additionally, the text contains numerous non-administrative region keywords such as "Public Security Bureau" and "Branch," making it impossible to directly use the raw text to meet the business system's requirements for standardized provincial and municipal codes. Currently, there is no dedicated data cleaning solution in the industry designed for this type of defective vehicle registration certificate issuing authority text, forcing the business side to still rely on manual verification and input of provincial and municipal information.

[0068] Among existing data cleaning methods, precise string matching algorithms (such as KMP and Boyer-Moore algorithms) can achieve high-precision text matching, but they have extremely high requirements for the completeness of the input text. When there are typos or missing characters in the text, the matching failure rate increases sharply. Fuzzy matching methods based on edit distance can tolerate a certain degree of character differences, but when processing long texts with a large number of irrelevant keywords, they have high computational complexity and are prone to producing incorrect matches. Text similarity calculation methods based on vector space models (such as TF-IDF combined with cosine similarity) can achieve semantic similarity calculation, but in short, highly structured matching scenarios such as administrative division names, they are prone to producing incorrect results of mismatched provinces and cities due to the lack of constraints on hierarchical relationships.

[0069] Furthermore, existing caching strategies are mostly applied to database query result caching or web page caching. In the field of text data cleaning, there is a lack of caching design specifically for the scenario of "target text to encoded mapping result," which cannot effectively reduce the repetitive computation overhead in high-concurrency scenarios. At the same time, stop word filtering strategies in existing technologies are mostly applied to the field of full-text search. Their stop word libraries are usually general-purpose stop word lists, lacking a dedicated stop word library design for specific non-administrative region keywords in the text of vehicle registration certificate issuing authorities. Moreover, updating the stop word library requires redeploying the system, lacking hot configuration capabilities.

[0070] In view of this, in order to overcome the shortcomings of related technologies, the address-oriented data cleaning method in this embodiment combines... Figure 1 In terms of overall design, it includes the following steps S110-S130.

[0071] Step S110: Standardize and preprocess the original text obtained by OCR recognition to obtain the core target text.

[0072] In the processing of vehicle registration certificate issuing authority text, the raw text obtained by OCR recognition often contains a large number of keywords that are not related to administrative divisions, as well as various invalid characters. This irrelevant information directly participates in subsequent matching and parsing, which can seriously interfere with matching accuracy. Therefore, before performing matching, it is necessary to perform standardized preprocessing on the raw text to eliminate interference factors.

[0073] As a preferred implementation, the above-mentioned standardized preprocessing specifically includes an effective character filtering step and a stop word truncation filtering step. Specifically, the original text is filtered for effective characters, filtering out non-Chinese characters and retaining only Chinese characters; based on a preset stop word library, the filtered text is subjected to keyword matching and truncation processing, deleting the matched stop words and their subsequent characters to obtain the core target text.

[0074] Specifically, the effective character filtering step is used to clean the original text obtained from OCR recognition. Since the effective information in the issuing authority text of the vehicle registration certificate consists entirely of Chinese characters (excluding minority characters), while the original text may contain irrelevant characters such as numbers, letters, special symbols, spaces, and invalid placeholders, these non-core characters can interfere with the subsequent matching process. Through character recognition and filtering, only the Chinese characters in the original text are retained, and all non-Chinese characters are filtered out to reduce the interference of non-core characters on the matching.

[0075] For example, if the original text obtained by OCR recognition is "冀A·12345河北省石家庄市公安局交通警察支队车辆管理所", after valid character filtering, "冀河北省石家庄市公安局交通警察支队车辆管理所" is obtained. Among them, the letters, numbers, symbols and spaces in "冀A·12345" are all filtered out, and only Chinese characters are retained.

[0076] The stop word dictionary truncation and filtering step is used to further process the filtered text. This step calls a dedicated stop word dictionary customized for the scenario of the issuing authority of the driving license, and adopts a sequential matching and truncation retention strategy. The system scans the stop word dictionary in a preset order, and the stop word dictionary stores high-frequency non-administrative-region keywords such as "public security bureau", "government", "detachment", "sub-bureau", "traffic police brigade", and "committee". During the scanning process, the filtered text is matched character by character in sequence. Once any stop word keyword is detected in the text, the system immediately performs a truncation operation, retains all character content before the stop word, and automatically deletes the stop word and all remaining text after the stop word. The above truncation operation does not distinguish the position of the stop word at the beginning, middle or end of the text, and the truncation is performed as soon as a match is detected. Through the above stop word filtering, the finally remaining pure administrative region-related text is the core target text, which serves as the only basis for subsequent hierarchical matching and parsing.

[0077] Continuing the above example, the filtered text is "冀河北省石家庄市公安局交通警察支队车辆管理所". When scanning the stop word dictionary, it is detected that "public security bureau" is located after "Hebei Province Shijiazhuang City", the system performs a truncation operation, retains all characters before "public security bureau" as "冀河北省石家庄市", and deletes "public security bureau" and all content after it. Finally, the core target text "冀河北省石家庄市" is obtained.

[0078] It should be noted that "冀" in the above example is the abbreviation of Hebei Province. In the subsequent matching, since the regional code table stores the standard administrative region name "Hebei Province" instead of the abbreviation "冀", "冀" will not be matched in the provincial left matching. However, since both "Hebei Province" and "Shijiazhuang City" are retained in the core target text, "Hebei Province" can be matched successfully during provincial left matching, and "Shijiazhuang City" can also be matched successfully during municipal left matching.

[0079] Further, if the core target text is "北京市公安局公安交通管理局车辆管理所", after being truncated by the stop word dictionary, the core target text "Beijing City" is obtained, so as to correctly match the municipality directly under the central government "Beijing City".

[0080] Furthermore, to enhance the flexibility and adaptability of the stop word library, a hot configuration mechanism is preferably adopted, including: the stop word library is centrally managed by a configuration center, supporting online addition, modification, or deletion of stop words; the configuration center pushes changes to the stop word library to the local service in real time; and the local service dynamically updates its locally cached stop word library upon receiving the push. Specifically:

[0081] The stop word library is centrally managed by a configuration center, which is independently deployed. Operations personnel can add, modify, or delete keywords and corresponding matching rules in the stop word library online through a visual interface, without writing code or restarting the system. After the configuration center completes the operation, it pushes the update command to the local service in real time via a message push mechanism (such as MQ messages or long-connection push). The local service listens for update notifications from the configuration center in real time, and automatically synchronizes the latest stop word library data and configuration rules upon receiving the update command. The changes take effect immediately after synchronization. A hot configuration mechanism ensures that the stop word library can quickly adapt to scenarios such as the addition and modification of keywords in the vehicle registration certificate issuing authority's text, guaranteeing the timeliness and flexibility of the filtering logic.

[0082] For example, when the business discovers that the new keyword "vehicle management office" appears in the text of the issuing authority of a vehicle registration certificate in a certain region, interfering with the matching, the operation and maintenance personnel can add "vehicle management office" to the stop word library through the configuration center's visual interface. The configuration center will immediately push the update to all local service nodes, and the service will take effect immediately without the need for shutdown or restart.

[0083] Through the above standardized preprocessing, the matching interference of non-administrative region keywords and irrelevant characters can be eliminated from the source.

[0084] After obtaining the standardized preprocessed core target text in step S110, it needs to be matched with standardized administrative region data. To improve the accuracy and efficiency of matching, as a preferred implementation, a cache query step is also included before executing the hierarchical matching strategy. This cache query step is used to improve parsing efficiency in high-concurrency scenarios, but its presence or absence does not affect the integrity and correctness of the core matching logic of this method.

[0085] In real-world business scenarios, the core target text from the same issuing authority may appear repeatedly (e.g., a large number of vehicles registering in the same region). Repeatedly performing a complete matching calculation on the same target text each time would waste significant computing resources and time. Therefore, the preprocessed core target text is used as the key to query the local cache.

[0086] The aforementioned local cache library uses a key-value pair storage structure, where the key is the core target text and the value is the encoding mapping result corresponding to that text (including the province / city name and its corresponding national standard code). The local cache library is preferably implemented based on a Redis in-memory database, and the cache time is configurable, with a default of 24 hours.

[0087] In practice, if a valid cache entry with the core target text as the key exists in the local cache library, it is determined that the cache has been hit. The encoding mapping result stored in the cache entry is read, and the read encoding mapping result is directly output as the structured parsing result, skipping all subsequent matching processes.

[0088] If no cache entry with the core target text as the key exists in the local cache, it is determined as a cache miss, and the hierarchical matching strategy in step S120 is executed. After the hierarchical matching parsing is completed, the final mapping result is written to the cache for quick querying of the same target text in subsequent transactions. The above cache query mechanism can avoid repeated text matching calculations and improve parsing efficiency in high-concurrency business scenarios.

[0089] For example, after parsing the first core target text "Hebei Province, Shijiazhuang City", the system uses "Hebei Province, Shijiazhuang City" as the key and stores the corresponding mapping result (province name: Hebei Province, province code: 130000, city name: Shijiazhuang City, city code: 130100) in a Redis cache for 24 hours. When the same "Hebei Province, Shijiazhuang City" appears again, the system directly reads the result from the cache without re-performing the matching calculation.

[0090] Step S120: Using the core target text, execute a hierarchical matching strategy to obtain the corresponding administrative region information. The hierarchical matching strategy prioritizes exact matching and uses text vector matching as a fallback.

[0091] The hierarchical matching strategy employs a layered processing logic that prioritizes precise matching and uses text vector matching as a fallback. Precise matching serves as the core matching method, while text vector matching acts as a supplementary method, effectively addressing issues such as typos and missing characters in OCR-recognized text.

[0092] The above hierarchical matching strategy includes the following steps S121-S122:

[0093] Step S121: Perform exact matching based on left character matching and perform association verification by level.

[0094] Specifically, the exact matching process based on left-side character matching is as follows:

[0095] (i) Provincial-level left matching. The core target text is matched with the provincial (level 1) administrative region names in the regional code table. Left matching means starting from the leftmost character of the target text and comparing it character by character with the administrative region name. A match is successful when the prefix of the target text is exactly the same as a certain provincial-level administrative region name.

[0096] It should be noted that the leftmost part of the core target text may contain provincial abbreviations such as "Ji" and "Jing", or may directly be full names such as "Hebei Province" and "Beijing City". Since the regional code table stores the standard full names of administrative regions, only full names can be matched successfully, and abbreviations will not be matched, which can effectively avoid ambiguous matching caused by abbreviations. In the subsequent text vector matching step, for cases where the provincial-level matching fails, the system can also effectively process text scenarios containing only abbreviations through semantic similarity calculation. Specifically, when the text to be matched contains only a provincial abbreviation (such as "Ji") without a full name, in the vector matching stage, the pre-trained vectorization model maps "Ji" and "Hebei Province" to adjacent vector space positions, and realizes semantic matching from the abbreviation to the full name by using cosine similarity calculation, without the need to additionally maintain an abbreviation-full name mapping table.

[0097] After the matching is successful, the provincial standard name and the corresponding national standard code are extracted, and the matched provincial name text is deleted from the core target text to generate a remaining text. This remaining text will be passed to the municipal-level matching step. When the provincial left matching fails, the exact matching process is interrupted, and the process proceeds to the switch judgment step of text vector matching described below.

[0098] For example, the core target text "Shijiazhuang City, Hebei Province" is left-matched with the provincial name "Hebei Province" in the regional code table, the prefix "Hebei Province" of "Shijiazhuang City, Hebei Province" is completely consistent with "Hebei Province", and the matching is successful. The provincial information (province name: Hebei Province, province code: 130000) is extracted, and the remaining text "Shijiazhuang City" is generated after deleting "Hebei Province".

[0099] In addition, if the core target text is "Shijiazhuang City Hebei Province" (this situation is relatively rare in actual OCR recognition, but exists in theory), then "Shijiazhuang City" does not match "Hebei Province" during left matching, the provincial-level matching fails, and the process proceeds to the text vector matching step.

[0100] (2) Municipal-level left matching. The remaining unmatched text generated after provincial-level matching is left-matched with the municipal (level 2) administrative region names in the regional code table. After the matching is successful, the municipal standard name and the corresponding national standard code are extracted. When the municipal-level left matching fails, the exact matching process is interrupted, and the process proceeds to the switch judgment step of text vector matching described below.

[0101] Continuing the above example, the remaining text "Shijiazhuang City" is left-matched with the municipal name "Shijiazhuang City" in the regional code table, which is completely consistent, and the matching is successful. The municipal information (city name: Shijiazhuang City, city code: 130100) is extracted.

[0102] (3) Level association verification. To ensure the absolute accuracy of the exact matching result, after both the provincial-level and municipal-level matching are successful, it is necessary to verify whether the superior code of the matched municipal code is completely consistent with the provincial code.

[0103] Continuing with the previous example, the superordinate code of the city-level code 130100 is 130000, which is completely consistent with the provincial code 130000. The verification passed, and the exact match was successful.

[0104] When the core target text is "Shijiazhuang City, Hebei Province", the OCR recognition mistakenly identifies "Shijiazhuang City" as "Shijiazhuang". The system matches the city name "Shijiazhuang" (assuming that the city name "Shijiazhuang" exists in the regional code table). However, the superior code of "Shijiazhuang" is not the Hebei Province code 130000. Therefore, the hierarchical association verification fails and the accurate matching is interrupted.

[0105] When the core target text is "Baoding City, Hebei Province", the superior code of the matched city-level code 130600 is 130000, which is consistent with the provincial code 130000, and the verification passes.

[0106] When the superior code of a city-level code is inconsistent with the provincial code, the precise matching process is interrupted, and the system enters the on / off judgment stage of text vector matching. In this case, the system will retain the provincial information that has been precisely matched (including the provincial name and code), and use this provincial information as a constraint in subsequent city-level vector matching. The system will only adopt the result if the superior code of the matched city-level candidate is consistent with the provincial code, thereby avoiding cross-provincial mismatches (e.g., avoiding mismatching "Zhengzhou City, Hebei Province" as "Zhengzhou City, Henan Province"). Furthermore, this application preferably records in detail the specific reasons for the verification failure, including but not limited to the matched provincial code, city code, the superior code corresponding to the city code, and key information such as the core target text, forming a standardized failure log for subsequent troubleshooting, matching rule optimization, and system maintenance.

[0107] After the above exact match fails, proceed to the text vector matching judgment step S122.

[0108] Step S122: If an exact match fails, determine whether the text vector matching switch is turned on.

[0109] When the text vector matching switch is turned on, text vector matching based on cosine similarity is performed, and hierarchical linkage verification is carried out; when the text vector matching switch is turned off or vector matching fails, an empty value is returned and the encoding and assembly stage is entered.

[0110] Specifically, text vector matching switch configuration items are stored in a separate configuration center, managed uniformly with dynamic configuration items such as the stop word library and cosine similarity threshold. The configuration center supports a visual interface for configuration, allowing operations personnel to operate directly without modifying code or restarting the business system. Once the configuration center completes the modification of the switch status (on / off), it immediately sends the latest configuration command to all business service nodes via a message push mechanism. Local services listen in real time and instantly load and apply the switch status, ensuring that configuration changes are synchronized to the runtime environment within milliseconds, without affecting the currently ongoing matching process.

[0111] For example, operations and maintenance personnel can switch the text vector matching switch from "off" to "on" through the configuration center interface. The configuration center will push the change to all service nodes through MQ messages. Each node will complete the status update within milliseconds, and subsequent requests will perform text vector matching according to the on state.

[0112] After an exact match fails, before proceeding to the text vector matching logic, the system first checks whether the text vector matching switch is enabled. If the switch is disabled, text vector matching is no longer performed, and an empty value is returned directly, proceeding to the encoding and assembly stage. Simultaneously, the system logs a warning message "Text vector matching switch disabled, exact match failed" and accumulates one match failure count, allowing maintenance personnel to view the match failure rate in real time through the monitoring platform, enabling timely assessment of whether the text vector matching switch needs to be enabled or the matching strategy adjusted.

[0113] In one preferred embodiment, after the text vector matching switch is turned on and an exact match fails, the text vector matching logic is entered. This logic uses text vectorization combined with a cosine similarity algorithm for inaccurate matching, designed to address issues such as typos and missing characters in OCR-recognized text.

[0114] Specifically, the text vector matching process is as follows:

[0115] First, word segmentation and vectorization are performed. Depending on the matching stage, the text to be matched is determined: if provincial-level matching fails, the core target text is used as the text to be matched; if provincial-level matching succeeds but city-level matching fails, the remaining unmatched text is used. The text to be matched is then processed using Chinese word segmentation, dividing it into several word sequences. These word sequences are then converted into fixed-dimensional numerical vectors using a pre-trained vectorization model. Simultaneously, the administrative region names corresponding to the corresponding levels in the regional code table are also pre-segmented, vectorized, and stored to improve real-time matching efficiency.

[0116] For example, if the core target text for OCR recognition is "Hebei Province, Shijiazhuang City" (where "Bei" is misidentified as "Bi"), the provincial left-hand matching will fail because "Hebei Province" and "Hebei Province" are not completely consistent. After entering text vector matching, the text to be matched, "Hebei Province, Shijiazhuang City," is segmented into word sequences ["Hebei Province", "Shijiazhuang City"], which are then converted into numerical vectors using a vectorization model. Simultaneously, the segmented word vector for the provincial name "Hebei Province" in the regional code table is pre-stored.

[0117] Next, cosine similarity calculation and candidate result filtering are performed. The cosine similarity between the numerical vector of the text to be matched and the corresponding administrative region name vector in the regional code table is calculated. The calculated result is compared with a preset cosine similarity threshold, and candidate results with a cosine similarity greater than the preset threshold are selected.

[0118] Continuing with the previous example, we calculate the cosine similarity between the vector of "Shijiazhuang City, Hebei Province" and the vector of "Hebei Province". Assuming the result is 0.95 (greater than the preset threshold of 0.80), then "Hebei Province" is included in the candidate provincial results.

[0119] Next, hierarchical linkage validation is performed. Hierarchical correlation validation is applied to the selected candidate results. Specifically, different validation logic is used based on the different result states of exact matches:

[0120] Scenario 1: When the provincial level matches successfully but the city level fails in the exact match, vector matching is performed only on the remaining unmatched city-level text. The cosine similarity between the city-level text and the names of all city-level administrative regions under that province in the regional code table is calculated. Candidate city-level results with similarity greater than or equal to the configured threshold are selected. Then, it is checked whether the parent code of the candidate city-level result is consistent with the provincial code of the exact match. If they are consistent, the match is successful, and the city-level name and code are extracted; otherwise, the match fails.

[0121] For example, the core target text is "Hebei Province Shijiazhuang City" ("Zhuang" was misidentified as "Zhuang"). The provincial-level "Hebei Province" was accurately matched. The cosine similarity between the remaining text "Shijiazhuang City" and "Shijiazhuang City" is 0.92 (threshold 0.80). The parent code of the candidate city-level result "Shijiazhuang City" is 130000, which is consistent with the provincial code 130000, so the match was successful.

[0122] In scenario two, if provincial-level matching fails but municipal-level matching succeeds in exact matching, then first perform provincial-level vector matching on the core target text, calculating its cosine similarity with all provincial-level administrative region names in the regional code table, and filtering out candidate provincial-level results. If municipal-level matching succeeds in exact matching, verify whether the encoding of the candidate provincial-level result is consistent with the superior encoding of the municipal-level result; if they are consistent, the matching is successful. If both provincial and municipal-level exact matching fail, first determine the corresponding municipal-level administrative region based on the candidate provincial-level results, then perform municipal-level vector matching on the core target text, filtering out municipal-level results with similarity greater than or equal to the configured threshold and hierarchical association; if the matching is successful, extract the corresponding provincial and municipal information; otherwise, the matching fails.

[0123] For example, the core target text is "Hebei Province Shijiazhuang City" (with "Bei" misidentified as "Bi"), resulting in a provincial-level match failure. However, the remaining text "Shijiazhuang City" achieved a precise city-level match. In the provincial-level vector matching, the cosine similarity between "Hebei Province" and "Hebei Province" is 0.95. The code 130000 for the candidate provincial result "Hebei Province" matches the parent code 130000 for the city-level result "Shijiazhuang City," indicating a successful match.

[0124] Scenario 3: If both provincial and municipal matching fail in exact matching, the corresponding municipal administrative region is first locked based on the candidate provincial results. Then, municipal vector matching is performed on the core target text to filter municipal results with similarity greater than or equal to the configured threshold and hierarchical association. If the match is successful, the corresponding provincial and municipal information is extracted.

[0125] For example, the core target text is "Hebei Province Shijiazhuang City" ("Bei" is misidentified as "Bi" and "Zhuang" as "Zhuang"), and both the provincial and municipal levels fail to match precisely. The system first determines the candidate province as "Hebei Province" through provincial vector matching, then narrows down the municipal level to all municipal administrative regions under the jurisdiction of Hebei Province, and then performs vector matching within the municipal level to determine the candidate municipal level as "Shijiazhuang City". After verifying that its superior code matches the code of Hebei Province, the match is successful.

[0126] Finally, the results are evaluated. If a valid result is found in the above steps, the corresponding provincial / municipal standard name and national standard code are extracted. If no valid result is found, an empty value is returned, and the encoding assembly stage begins.

[0127] The aforementioned cosine similarity threshold preferably supports dynamic configuration. This threshold can be dynamically adjusted based on the quality of the OCR-recognized text: when there are severe typos or missing characters, the threshold can be appropriately lowered to improve recall; when the text quality is good, the threshold can be increased to improve accuracy. Furthermore, the threshold can be adjusted in real-time according to the matching accuracy requirements of the current business scenario. The word segmentation algorithm preferably employs dedicated word segmentation logic adapted to government texts to improve the accuracy of administrative region name segmentation.

[0128] Among them, the word segmentation algorithm preferably adopts a dedicated word segmentation logic adapted to government documents. Unlike general word segmentation algorithms, the dedicated word segmentation logic for government documents can correctly handle words with specific administrative division structures, such as "Shijiazhuang City" and "Hohhot City", avoiding the incorrect segmentation of "Shijiazhuang" into "Shijia" and "Zhuang", thus improving the accuracy of word segmentation of administrative division names.

[0129] After completing precise matching or text vector matching in step S120, the matched administrative region information needs to be converted into standardized national standard codes and output in a structured form for direct use by downstream business systems, i.e., step S130 is executed.

[0130] Step S130: Based on the administrative region information obtained through matching, map the corresponding national standard code from the regional code table and output the structured parsing result.

[0131] Specifically, a structured regional code table is established as the standardized core data source for encoding mapping. This regional code table stores the standard names, national standard codes, and hierarchical relationships of provincial and municipal administrative divisions in a structured manner. The regional code table covers all legally mandated administrative divisions, including provinces, autonomous regions, municipalities, and special administrative regions.

[0132] Municipalities directly under the central government are uniformly categorized and managed at the provincial level (Level 1), without their own city-level (Level 2) data. The parsing results are directly obtained using the municipal standard name and national standard code. In this case, the "Issuing City Standard Name" field in the structured output is filled with the name of the municipality (same as the provincial name), and the "Issuing City National Standard Code" field is filled with the provincial national standard code of the municipality, to maintain the integrity and consistency of the output structure and facilitate unified processing by downstream business systems. Autonomous regions are managed at the provincial level (Level 1), and their subordinate cities (prefectures, leagues) are managed at the city level (Level 2). Special administrative regions are treated as a separate provincial level (Level 1), without city-level divisions. The above settings are adapted to various special administrative division scenarios.

[0133] The regional code table preferably adopts a dual synchronization mechanism combining real-time database synchronization and API call fallback, with the synchronization period set to once per hour. During the synchronization process, core data is preferentially synchronized through direct database connection. If the database synchronization fails, the API call method is automatically triggered to retry synchronization, ensuring that the regional code table data remains consistent with the core system. The update of the regional code table and the hot configuration of the stop word list are independent of each other and do not affect each other. When the national administrative division adjustment leads to the addition or change of administrative region names in the regional code table, the stop word list does not need to be adjusted synchronously, because the stop word list is only used to filter functional keywords such as "Public Security Bureau" and "Detachment", and does not involve the administrative region names themselves. If the newly added administrative region name contains a keyword overlapping with the stop word list (for example, a certain region name contains the two characters "Public Security"), an exception rule for the region name needs to be added in the stop word list to avoid erroneous filtering. This exception rule is also added online through the hot configuration mechanism of the configuration center and takes effect immediately.

[0134] For example, when the state adjusts the administrative divisions (such as adding new county-level cities, changing administrative division codes, etc.), after the core system code value library is updated, the regional code table automatically completes data update through the synchronization mechanism within one hour to ensure the accuracy of code mapping.

[0135] In addition, code mapping and structured output are performed according to the matching result. Specifically, the successfully matched standard names of provinces and cities are mapped one-to-one with the national standard codes in the regional code table, and structured parsing results are assembled.

[0136] The core output fields include: standard name of the certificate-issuing province, national standard code of the certificate-issuing province, standard name of the certificate-issuing city, national standard code of the certificate-issuing city, matching type (exact match or text vector match), and matching confidence (the cosine similarity value for text vector matching, and 1 for exact match). If the matching fails (no valid result is obtained through both exact match and text vector match), the province / city code and name fields return null values, and a matching failure identifier is output at the same time, which is convenient for downstream business systems to perform manual intervention processing.

[0137] In addition, the above structured parsing results are output to downstream business systems through a standardized interface, realizing seamless docking with systems such as auto finance application, used car evaluation, and vehicle management.

[0138] The embodiment of the second aspect of the present application provides a data cleaning system for address information, such as Figure 2 shown in the figure, comprising a preprocessing module 410, a layered matching module 420, and a code mapping module 430.

[0139] The preprocessing module 410 is used to perform standardized preprocessing on the original text obtained by OCR recognition to obtain the core target text. The hierarchical matching module 420 is used to execute a hierarchical matching strategy using the core target text to match and obtain the corresponding administrative region information. The encoding mapping module 430 is used to map the corresponding national standard code from the regional code table according to the matched administrative region information and output a structured parsing result.

[0140] The hierarchical matching strategy adopts a strategy of prioritizing exact matching and using text vector matching as a fallback.

[0141] The specific implementation methods and technical details of the above modules correspond to the corresponding steps in the address-oriented data cleaning method described in the first aspect, and will not be repeated here.

[0142] The above descriptions are merely some embodiments of this application and are not intended to limit this application. The technical features or structures in the foregoing different embodiments can be arbitrarily combined to form other specific technical solutions as needed. For those skilled in the art, this application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of the claims of this application.

Claims

1. A data cleaning method oriented towards address information, characterized in that, The data cleaning method includes: The original text obtained by OCR recognition is standardized and preprocessed to obtain the core target text; Using the core target text, a hierarchical matching strategy is executed to obtain the corresponding administrative region information; Based on the administrative region information obtained through matching, the corresponding national standard code is mapped from the regional code table, and a structured parsing result is output. The hierarchical matching strategy adopts a strategy of prioritizing exact matching and using text vector matching as a fallback.

2. The data cleaning method for address information according to claim 1, characterized in that, The standardized preprocessing includes: The original text is filtered for valid characters, removing non-Chinese characters and retaining only Chinese characters. Based on a pre-set stop word library, the filtered text is subjected to keyword matching and truncation, and the matched stop words and their subsequent characters are deleted to obtain the core target text.

3. The data cleaning method for address information according to claim 2, characterized in that, The stop word library adopts a hot configuration mechanism, including: The stop word library is centrally managed by the configuration center and supports adding, modifying or deleting stop words online. The configuration center pushes changes to the stop word library to the local service in real time; The local service dynamically updates the local cached stop word library after receiving push notifications.

4. The data cleaning method for address information according to claim 1, characterized in that, Before executing the hierarchical matching strategy, the data cleaning method further includes performing a cache query, and the cache query includes: Using the core target text as the key, query the local cache library, wherein the local cache library adopts a key-value pair storage structure, the key is the core target text, and the value is the encoding mapping result corresponding to the text; When a valid cache entry with the core target text as the key exists in the local cache library, it is determined as a cache hit. The encoding mapping result stored in the cache entry is read, and the read encoding mapping result is directly output as the structured parsing result. If no cache entry with the core target text as the key exists in the local cache library, it is determined as a cache miss, and the hierarchical matching strategy continues to be executed.

5. The data cleaning method for address information according to claim 1, characterized in that, The hierarchical matching strategy includes: Perform exact matching based on left-hand character matching, and perform association validation by level; When an exact match fails, check whether the text vector matching switch is enabled. Specifically, when the text vector matching switch is turned on, text vector matching based on cosine similarity is performed, and hierarchical linkage verification is carried out. If the text vector matching switch is off or vector matching fails, an empty value is returned.

6. The data cleaning method for address information according to claim 5, characterized in that, The execution of precise matching based on left-hand character matching includes: The core target text is left-matched with the provincial administrative region names in the regional code table. After a successful match, the provincial information is extracted, and the matched provincial names in the core target text are deleted to generate the remaining text. The remaining text is left-matched with the city-level administrative region names in the regional code table. If the match is successful, the city-level information and the corresponding city-level code are extracted. Verify whether the superior code of the city-level code obtained by matching is consistent with the provincial code.

7. The data cleaning method for address information according to claim 5, characterized in that, The text vector matching based on cosine similarity includes: The text to be matched is processed by Chinese word segmentation, which divides the text into several word sequences, and the word sequences are converted into fixed-dimensional numerical vectors through a pre-trained vectorization model. Calculate the cosine similarity between the numerical vector and the corresponding administrative region name vector in the regional code table; The cosine similarity calculation result is compared with the preset cosine similarity configuration threshold. Candidate results with a cosine similarity greater than the preset cosine similarity configuration threshold are selected, and the selected candidate results are subjected to hierarchical linkage verification.

8. The data cleaning method for address information according to claim 7, characterized in that: The cosine similarity threshold supports dynamic configuration and uses different cosine similarity thresholds based on the OCR-recognized text quality score: and / or, The cosine similarity threshold can be dynamically configured and adjusted in real time according to the matching accuracy requirements of the current business scenario.

9. The data cleaning method for address-oriented information according to any one of claims 1 to 8, characterized in that: The regional code table stores the standard names, national standard codes, and hierarchical relationships of provincial and municipal administrative divisions in a structured manner. Municipalities and special administrative regions are managed at the provincial level and are not divided into city levels, while autonomous regions are managed at the provincial level and include data on their subordinate city levels.

10. A data cleaning system for address information, characterized in that, The data cleaning system includes: The preprocessing module is used to standardize and preprocess the raw text obtained by OCR recognition to obtain the core target text; The hierarchical matching module is used to execute a hierarchical matching strategy using the core target text to match and obtain the corresponding administrative region information; The encoding mapping module is used to map the corresponding national standard code from the regional code table based on the matched administrative region information, and output a structured parsing result; The hierarchical matching strategy adopts a strategy of prioritizing exact matching and using text vector matching as a fallback.

Images