Non-standard place name address data cleaning method and device based on word segmentation pattern matching

Through the method based on word participle pattern matching, errors and incomplete information in place name address data cleaning are solved, standardization and accuracy of place name address data are realized, and the circulation and application of geographic information data are promoted.

CN114647639BActive Publication Date: 2024-11-19CHINA UNIV OF GEOSCIENCES (WUHAN)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210277639.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-21
Publication Date
2024-11-19
Estimated Expiration
2042-03-21

AI Technical Summary

Technical Problem

The prior art is difficult to effectively clean and standardize place name address data, resulting in errors and incomplete information, affecting the accuracy and application of geographic information data.

Method used

Using a method based on word segmentation pattern matching, the word segmentation model is obtained and trained, indexes are constructed, and the geographic information database and the internal address database of the organization are matched, and the place name address data is separated and standardized.

Benefits of technology

It improves the accuracy of place name address data, realizes the standardization of place name address data, promotes the circulation and application of geographic information data, and reduces the losses caused by information asymmetry.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114647639B_ABST
    Figure CN114647639B_ABST
Patent Text Reader

Abstract

The present invention provides a method and device for cleaning non-standard place name and address data based on word segmentation pattern matching. The standard place name and address are searched in the database through word segmentation pattern matching. If there is an internal address database of the organization, the secondary address classification result is obtained by matching the internal address database. The secondary address classification result is combined with the previous standard place name and address to obtain the final result, otherwise the standard place name and address obtained previously is used as the final result. The features of text annotation in the Chinese context are learned through deep learning, and the part-of-speech tagging result is output through the word segmentation model after the data to be queried is obtained to obtain the word segmentation result, and the national standard administrative division elements and the internal address elements of the point of interest are separated, and then the database related to the point of interest is connected for secondary cleaning. The present invention converts non-standard place name and addresses with a large number of errors, confusing names, incomplete information and other problems into standard place name and address through data cleaning, thereby improving the accuracy of place name and address.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a method and device for cleaning non-standard place name and address data based on word segmentation pattern matching. Background Art

[0002] With the popularization of information technology, the amount of data generated by humans is growing at an exponential rate. Such a large amount of data requires new methods to be managed. In order to support correct decision-making, the managed data must be reliable, error-free, and accurately reflect the actual situation of organizations, enterprises and institutions, so as to improve data quality, achieve extensive data sharing, and ultimately maximize data value. Place name and address data cleaning is of great significance to the informatization construction of various enterprises and institutions.

[0003] Place name and address data are complex and numerous areas have been replanned and rebuilt. This phenomenon has caused more and more stock address data in the original system to be inconsistent with the actual address. In addition, due to some historical reasons, the existing address data may contain a large number of errors, confusing names, incomplete information, etc., which makes it extremely difficult to clean and standardize them. And because the stock addresses are not structured data, it also has an adverse impact on the construction of various analysis and decision support systems based on address data. It is of great practical significance to use pattern matching technology based on word library segmentation to clean the existing place name and address information.

[0004] In order to build a universal geographic information data system, it is necessary to preliminarily clean the non-standard place name and address data input by different users from all walks of life. The process of place name and address data information processing includes word segmentation, semantic annotation, matching, replacement and other processes. Currently, commonly used algorithms include the TF-IDF algorithm based on word frequency and inverse text frequency matching corresponding documents, and the forward and reverse matching algorithms for word segmentation methods. However, they are highly dependent on the corpus, and different corpora are prone to different word segmentation results. In addition, since there may be unregistered words in the uncleaned data, it is easier to make mistakes during matching. Summary of the invention

[0005] The main technical problem solved by the present invention is how to convert address data that may contain a large number of errors, confusing names, incomplete information, etc. into standard place name and address data with a higher accuracy through data cleaning means, and apply it in a general geographic information data system.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] According to one aspect of the present invention, the present invention provides a method for cleaning non-standard place name and address data based on word segmentation pattern matching, comprising the following steps:

[0008] S1: Obtain a preset word segmentation model and perform model training. After the training is completed, a trained word segmentation model is obtained;

[0009] S2: After obtaining the data to be queried and preprocessing it, the trained word segmentation model is input, and then the national standard administrative division elements and the internal address elements of the point of interest are output through the classifier;

[0010] S3: build the first index;

[0011] S4: index matching the national standard administrative division elements in the geographic information database according to the first index to obtain the address classification result;

[0012] S5: Determine whether there is an internal address database of the organization. If so, proceed to step S6; otherwise, proceed to step S9;

[0013] S6: construct a second index;

[0014] S7: when receiving a query request for an internal address element of a point of interest, index matching is performed on the internal address element of the point of interest in an internal address database of the organization according to the second index to obtain a secondary address classification result;

[0015] S8: combining the address classification result and the secondary address classification result as the final result of the standard place name address, and ending the process;

[0016] S9: The address classification result is used as the final result of the standard place name address, and the process ends.

[0017] Preferably, step S1 of training a preset word segmentation model includes:

[0018] S1.1: Obtain text from geographic information database;

[0019] S1.2: Use character filters to remove full-width or half-width non-text symbols in the text;

[0020] S1.3: Eliminate duplicate data;

[0021] S1.4: Manually mark some parts of speech;

[0022] S1.5: Replace synonyms to obtain preprocessed text;

[0023] S1.6: The preset word segmentation model is trained through the preprocessed text. After the training is completed, a trained word segmentation model is obtained.

[0024] Preferably, in step S1, the word segmentation model is a BiGRU+CRF word segmentation model;

[0025] The BiGRU+CRF word segmentation model is divided into five layers, which are, in the order of forward propagation: input layer, word vector layer, BiGRU layer, CRF layer and output layer. After the model training is completed, the word segmentation result and word segmentation type are determined according to the output result of the CRF layer, and the national standard administrative division elements and internal address elements of the point of interest are output through the classifier.

[0026] Preferably, the step of preprocessing the query data in step S2 includes:

[0027] S2.1: Use character filters to remove full-width or half-width non-text symbols in the query data;

[0028] S2.2: Integrate a large number of query requests into query streams, remove duplicate query data, and allocate computing nodes;

[0029] S2.3: Regularize the query data to obtain preprocessed data.

[0030] Preferably, the step of constructing the first index in step S3 includes:

[0031] S3.1: Obtain national standard administrative divisions and POI addresses from the geographic information database;

[0032] S3.2: Construct a global search tree based on pinyin according to the national standard administrative divisions and POI address data, and construct multiple parallel first index libraries according to word segmentation classes, where each index points to a node on the tree.

[0033] Preferably, the step of constructing the second index in step S6 includes: constructing the second index based on the internal address database of the organization.

[0034] Preferably, the national standard administrative division elements and internal address elements of the point of interest output in step S2 are sent one-to-one to the geographic information database and the internal address database of the institution for index matching, wherein the index matching in the internal address database of the institution and the index matching in the geographic information database are completed and meet the internal authority requirements of the platform and the permission of the point of interest institution before they can be carried out.

[0035] Preferably, the geographic information database has a complete data structure, and the internal address database of the organization is provided by the point of interest organization or constructed with the help of the platform and can only be used after obtaining the permission of the point of interest organization.

[0036] Preferably, the elements for index matching include national standard administrative division elements and internal address elements of points of interest obtained by inference and re-division from the trained word segmentation model, and also include elements obtained by matching through a synonym table.

[0037] According to the second aspect of the present invention, the present invention also provides a non-standard place name and address data cleaning device based on word segmentation pattern matching, comprising the following modules:

[0038] The model training module is used to obtain the preset word segmentation model and perform model training. After the training is completed, the trained word segmentation model is obtained;

[0039] The feature classification module is used to obtain the data to be queried and pre-process it, input the trained word segmentation model, and then output the national standard administrative division features and the internal address features of the point of interest through the classifier;

[0040] A first index building module, used to build a first index;

[0041] An address classification module is used to index and match the national standard administrative division elements in the geographic information database according to the first index to obtain an address classification result;

[0042] An internal address database determination module is used to determine whether there is an internal address database of the organization. If so, the module enters the second index construction module; otherwise, the module enters the second standard place name address output module;

[0043] A second index building module, used for building a second index;

[0044] A secondary address classification module, for, when receiving a query request for an internal address element of a point of interest, index matching the internal address element of the point of interest in an internal address database of the organization according to a second index, to obtain a secondary address classification result;

[0045] The first standard place name address output module is used to combine the address classification result and the secondary address classification result as the final result of the standard place name address and end the process;

[0046] The second standard place name address output module is used to use the address classification result as the final result of the standard place name address and end the process.

[0047] The beneficial effects brought by the technical solution provided by the present invention are:

[0048] (1) Search for standard place name addresses in the database through word segmentation pattern matching. If there is an internal address database of the organization, the secondary address classification result is obtained by matching the internal address database. The secondary address classification result is combined with the previous standard place name address to obtain the final result. Otherwise, the previously obtained standard place name address is used as the final result. Through deep learning, the features of text annotation in the Chinese context are learned. After obtaining the data to be queried, the word segmentation result is output through the word segmentation model to obtain the word segmentation result. It is easy to separate the national standard administrative division elements and the internal address elements of the point of interest, and then access the database related to the point of interest for secondary cleaning. Non-standard place name addresses with a large number of errors, confusing names, incomplete information, etc. are converted into standard place name addresses through data cleaning, which improves the accuracy of place name addresses.

[0049] (2) By building a universal geographic information database system, it is beneficial to facilitate the flow of data between different applications, helping users of applications that have been equipped with this technology to better understand geographic information and reduce losses caused by information asymmetry. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0051] Figure 1 The overall flow chart of the non-standard place name and address data cleaning method based on word segmentation pattern matching of the present invention;

[0052] Figure 2 It is the abstract structure of the BiGRU+CRF network model of the present invention;

[0053] Figure 3 The structure of the three-pronged phonetic tree index of the present invention and the positions of the words "university" and "China" in the tree;

[0054] Figure 4 The figure is a general structural diagram of the non-standard place name and address data cleaning device based on word segmentation pattern matching of the present invention. DETAILED DESCRIPTION

[0055] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0056] like Figure 1 As shown, this embodiment provides a method for cleaning non-standard place name and address data based on word segmentation pattern matching, and its implementation is divided into three steps: model training, index construction and data query.

[0057] Model training follows the following process

[0058] 1. Data preprocessing

[0059] The main purpose is to process some obviously unreasonable or useless parts of the data, including the following steps:

[0060] (1) Use character filters to remove full-width or half-width non-text symbols such as “,” and “?” in the text.

[0061] (2) Eliminate duplicate data

[0062] (3) Manually mark some parts of speech.

[0063] (4) Synonymous word replacement.

[0064] 2. Training of BiGRU+CRF word segmentation model

[0065] Based on the deep learning method, the conditional field model is introduced through the neural network architecture with consistent encoding format to reduce the difficulty of annotation and improve the word segmentation ability. Its model structure is as follows Figure 2 As shown:

[0066] The model is divided into five layers. In the order of forward propagation, they are:

[0067] (1) Input layer: Since the input layer is of fixed length, it is necessary to determine the length of the input layer by analyzing the length of possible words.

[0068] (2) Word embedding layer: Since neural networks can only process numbers, it is necessary to convert characters into unified word embeddings, that is, to establish a mapping from words to vectors.

[0069] (3) BiGRU layer: used to learn the state transfer matrix of word sequences. Compared with BiRNN, it is less prone to gradient vanishing or gradient exploding. Compared with BiLSTM, its model training speed is faster and the effect is similar.

[0070] (4) CRF layer: used to learn the state transition probability matrix of different part-of-speech tagging sequences to predict the correct part-of-speech sequence.

[0071] (5) Output layer: Output the final tag sequence, and the word segmentation result can be obtained based on the tag sequence. The basic word position tag set used in this method is as follows:

[0072] Table 1 Basic word tagging set

[0073]

[0074] After the model training is completed, the segmentation results and segmentation types can be determined directly based on the output results of the CRF layer.

[0075] The smallest unit of public geographic information data currently available is the point of interest (POI). The final word segmentation results are divided into two categories through the classifier: the internal address of the point of interest (POISubArea) and the national standard administrative division address (including POI and words such as province, city, district and county), and then the word segmentation results are sent to the data stream for further processing.

[0076] Index building follows the following process:

[0077] 1. Synonym configuration

[0078] Synonyms often appear in Chinese, including abbreviations, aliases, old names, traditional Chinese characters, etc. For these words, an easily expandable synonym table relational database is constructed. For example, "China University of Geosciences-China University of Geosciences" is a synonym relationship. For synonyms, it is not necessary to directly replace them, but to add them to the data stream for retrieval.

[0079] 2. Pinyin B-Trigraph dictionary tree

[0080] A pinyin ternary tree is constructed for storage of the global national standard geographic administrative division data, and an index library is constructed for specific word segmentation based on part-of-speech classification.

[0081] Take the query "China / university" as an example.

[0082] In the word segmentation results, the word segmentation is graded according to the order of the word position tag set (which is also the order of administrative divisions). "China" is tagged as the word "country", which is closer to the root in the tree and has a higher grade. "University" is tagged as the word "poi", which is closer to the leaf node in the tree and has a lower grade.

[0083] Several parallel index databases are constructed based on the word parts of speech, and each database contains several pointers from specific words to the global dictionary tree.

[0084] During the query process, we start from the lowest-level word in all word segmentation streams and search toward the root node, and continuously add the words stored in the node to the result to form a standard place name. If we encounter a higher-level word that has been segmented in this query, the score of this result will be increased by 1, and finally the results will be sorted according to the score.

[0085] like Figure 3 As shown, three leaf nodes are found from the index of "university". After upward query, three results "China University of Geosciences", "China University of Metrology" and "Peking University" are obtained. The first two words have the same score because they both pass through the node "China", and the last word has a lower score because it only passes through one node.

[0086] The data query process follows the following process

[0087] 1. Data preprocessing

[0088] (1) Use character filters to remove full-width or half-width non-text symbols such as “,” and “?” in the text.

[0089] (2) Integrate a large number of query requests into query streams, remove duplicate data, and allocate computing nodes.

[0090] (3) Data regularization.

[0091] 2. Participle

[0092] Use the previously trained word segmentation model to segment the text and obtain word segmentation annotations. Then, according to the word type, the word group is sent to different word segmentation streams.

[0093] 3. Data reprocessing

[0094] After obtaining the set of internal address elements of the point of interest and national standard administrative division elements through the word segmentation results, the data is processed as follows:

[0095] (1) Check the stop word list and remove stop words.

[0096] (2) Eliminate repeated words.

[0097] (3) All Chinese characters are converted into pinyin, and separators are inserted between the pinyin.

[0098] After completing the above processing, the separated internal address elements of the point of interest are not used for the time being. First, the national standard administrative division elements are sent to the standard geographic information database constructed in the previous steps and stored on the service platform for query. Select the standard data with the smallest edit distance in the query result, and then check the query authority. If the authority is sufficient, query whether there is an internal address database for the point of interest on the geographic information sharing data platform. If there is no internal address database for the point of interest, the standard data is returned directly, otherwise it is combined with the data returned by the internal address database of the point of interest as the final result output.

[0099] The beneficial effect of the above scheme is that it separates the highly standardized national standard administrative division element data and POI data from the less standardized point of interest internal address element data. This allows you to spend more time training standard geographic data while separating the flexible point of interest internal address element data indexing and query functions into a plug-and-play function, thus facilitating data docking.

[0100] As an optional implementation, refer to Figure 4 This embodiment provides a non-standard place name and address data cleaning device based on word segmentation pattern matching, including:

[0101] Model training module 1 is used to obtain a preset word segmentation model and perform model training. After the training is completed, a trained word segmentation model is obtained;

[0102] The element classification module 2 is used to obtain the data to be queried and input the trained word segmentation model after preprocessing, and then output the national standard administrative division elements and the internal address elements of the point of interest through the classifier;

[0103] A first index building module 3, used to build a first index;

[0104] The address classification module 4 is used to index and match the national standard administrative division elements in the geographic information database according to the first index to obtain the address classification result;

[0105] Internal address database determination module 5, used to determine whether there is an internal address database of the organization, if so, enter the second index construction module; otherwise, enter the second standard place name address output module;

[0106] A second index building module 6, used to build a second index;

[0107] The secondary address classification module 7 is used for, when receiving a query request for an internal address element of a point of interest, index matching the internal address element of the point of interest in an internal address database of the organization according to a second index to obtain a secondary address classification result;

[0108] The first standard place name address output module 8 is used to combine the address classification result and the secondary address classification result as the final result of the standard place name address and end the process;

[0109] The second standard place name address output module 9 is used to use the address classification result as the final result of the standard place name address and end the process.

[0110] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0111] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order and these words may be interpreted as identifiers.

[0112] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for cleaning non-standard place name and address data based on word segmentation pattern matching, characterized in that: The steps include: S1: Obtain a preset word segmentation model and perform model training. After the training is completed, a trained word segmentation model is obtained; S2: After obtaining the data to be queried and preprocessing it, the trained word segmentation model is input, and then the national standard administrative division elements and the internal address elements of the point of interest are output through the classifier; S3: build the first index; S4: index matching the national standard administrative division elements in the geographic information database according to the first index to obtain the address classification result; S5: Determine whether there is an internal address database of the organization. If so, proceed to step S6; otherwise, proceed to step S9; S6: construct a second index; S7: when receiving a query request for an internal address element of a point of interest, index matching is performed on the internal address element of the point of interest in an internal address database of the organization according to the second index to obtain a secondary address classification result; S8: combining the address classification result and the secondary address classification result as the final result of the standard place name address, and ending the process; S9: The address classification result is used as the final result of the standard place name address, and the process ends.

2. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: Step S1 is a step of training a preset word segmentation model, including: S1.1: Obtain text from geographic information database; S1.2: Use character filters to remove full-width or half-width non-text symbols in the text; S1.3: Eliminate duplicate data; S1.4: Manually mark some parts of speech; S1.5: Replace synonyms to obtain preprocessed text; S1.6: The preset word segmentation model is trained through the preprocessed text. After the training is completed, a trained word segmentation model is obtained.

3. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: In step S1, the word segmentation model is a BiGRU+CRF word segmentation model; The BiGRU+CRF word segmentation model is divided into five layers, which are, in the order of forward propagation: input layer, word vector layer, BiGRU layer, CRF layer and output layer. After the model is trained, the word segmentation result and word segmentation type are determined according to the output result of the CRF layer, and the national standard administrative division elements and internal address elements of the point of interest are output through the classifier.

4. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: The step of preprocessing the query data in step S2 includes: S2.1: Use character filters to remove full-width or half-width non-text symbols in the query data; S2.2: Integrate a large number of query requests into query streams, remove duplicate query data, and allocate computing nodes; S2.3: Regularize the query data to obtain preprocessed data.

5. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: The step of constructing the first index in step S3 includes: S3.1: Obtain national standard administrative divisions and POI addresses from the geographic information database; S3.2: Construct a global search tree based on pinyin according to the national standard administrative divisions and POI address data, and construct multiple parallel first index libraries according to word segmentation classes, where each index points to a node on the tree.

6. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: The step of constructing the second index in step S6 includes: constructing the second index according to the internal address database of the organization.

7. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: The national standard administrative division elements and internal address elements of the points of interest output in step S2 are sent one by one to the geographic information database and the internal address database of the institution for index matching. The index matching in the internal address database of the institution and the index matching in the geographic information database are completed and meet the internal authority requirements of the platform and the permission of the point of interest institution before they can be carried out.

8. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: The geographic information database has a complete data structure. The internal address database of the organization is provided by the point of interest organization or built with the help of the platform and can only be used after obtaining the permission of the point of interest organization.

9. The method for cleaning non-standard place name and address data based on word segmentation pattern matching according to claim 1, characterized in that: The elements for index matching include national standard administrative division elements and internal address elements of points of interest obtained by inference and re-division from the trained word segmentation model, as well as elements obtained by matching through the synonym table.

10. A non-standard place name and address data cleaning device based on word segmentation pattern matching, characterized in that: include: The model training module is used to obtain the preset word segmentation model and perform model training. After the training is completed, the trained word segmentation model is obtained; The feature classification module is used to obtain the data to be queried and pre-process it, input the trained word segmentation model, and then output the national standard administrative division features and the internal address features of the point of interest through the classifier; A first index building module, used to build a first index; An address classification module is used to index and match the national standard administrative division elements in the geographic information database according to the first index to obtain an address classification result; An internal address database determination module is used to determine whether there is an internal address database of the organization. If so, the module enters the second index construction module; otherwise, the module enters the second standard place name address output module; A second index building module, used for building a second index; A secondary address classification module, for, when receiving a query request for an internal address element of a point of interest, index matching the internal address element of the point of interest in an internal address database of the organization according to a second index, to obtain a secondary address classification result; The first standard place name address output module is used to combine the address classification result and the secondary address classification result as the final result of the standard place name address and end the process; The second standard place name address output module is used to use the address classification result as the final result of the standard place name address and end the process.