A method for address extraction and standardization based on pre-training

Abstract

The present invention relates to address extraction, and in particular to an address extraction and standardization method based on pre-training, which comprises collecting corpus containing address information and pre-training a model; fine-tuning the pre-training model through a semi-supervised self-learning mode based on an enhanced address corpus, and using the fine-tuned model to perform place name recognition; performing address correction based on a self-updating and self-maintaining dictionary; and performing address normalization based on a multi-head attention mechanism generation model. The technical solution provided by the present invention can effectively overcome the defects of the prior art that the cost of annotated corpus is high and the extracted addresses are not standard enough.

Description

Technical Field

[0001] The present invention relates to address extraction, and in particular to an address extraction and standardization method based on pre-training. Background Art

[0002] The names of specific entities with specific scope, characteristics and directions gradually formed by certain social groups are called place names, which are unique language symbols. Place names are people's descriptions of the specific spatial location of specific objects. Because place names are the names of regional or landmark locations, they have specific structures, such as hierarchical structures. At the same time, because they are human beings' abstract representations of spatial locations or regions, they are specific language expressions formed by people's brain processing of geographical coordinates. Therefore, just like writing articles, this expression also presents the characteristics of thousands of faces, as shown in Table 1.

[0003] Human activities are closely related to addresses, and address information plays an increasingly important role in people's lives. For individuals, the registration and management of information such as native place, birthplace, and current residence address, the entry of origin and destination for taxi rides, express delivery, and travel navigation, etc.; for the government, the management of physical location information such as housing transactions and municipal facilities, and the rapid and accurate reporting of addresses when illnesses, dangers, or accidents occur are all closely related to address information. Due to the widespread use of address information, almost all walks of life are collecting address information, but there is no standard format for address information, so the address information obtained is complex and diverse; in addition, my country has a vast territory, many and complex government management departments, and there is a certain lag in address planning while urban construction is developing rapidly, which leads to a large number of abbreviations, aliases, popular names, irregular writing, and disordered structures in the address, and the description of the same address is inconsistent, incomplete, and non-standard. In addition, due to the characteristics of Chinese addresses, there are no separators, homophones, and they are easily missing or redundant, which makes it more difficult for us to extract and standardize address information.

[0004] At present, the extraction methods of Chinese addresses mainly include address dictionary, rule-based matching, statistics-based and deep learning-based entity recognition methods. The method of using address dictionary matching is relatively simple, but the disadvantages are also obvious. It cannot match place names that are not in the dictionary or are ambiguous. It can only extract existing and accurately described place names. The difficulty of dictionary construction is also relatively large, and the update and maintenance of the dictionary also requires a lot of manpower and material resources.

[0005] Rule-based matching generally has forward matching or reverse matching methods, but the construction of rules depends on specific regions and can only be constructed for specific regions. The constructed rules are difficult to apply to other regions and have poor universality. In addition, as the scope of the region expands, more rules need to be maintained, and the later maintenance cost is very high and prone to errors.

[0006] The statistical method can get rid of the dependence on the dictionary and extract the address by counting the frequency of co-occurring words. Frequent co-occurring words in the sentence can relatively reflect the confidence of their combination into the address. The statistical method often uses machine learning algorithm models such as N-gram model, HMM (Hidden Markov Model), CRF (Conditional Random Field), SVM (Support Vector Machine Model), Maximum Entropy Model, etc. This method is more dependent on feature settings. It is not easy to converge if there are too few features, and it is easy to overfit if there are too many features.

[0007] Commonly used methods based on deep learning are ELMO and Bi-LSTM+CRF, which have achieved good results in address extraction. However, the number of parameters in deep learning models is exponentially larger than that of original machine learning models. Larger model parameters mean that more corpus is needed for model training. However, supervised learning requires annotated corpus, and annotating corpus will incur huge time and labor costs, especially for some professional fields, where annotation is more difficult.

[0008] In order to solve the above problems, BERT+BiLSTM+CRF is generally used to extract place names. BERT is used based on the pre-trained model to perform fine-tuning on specific tasks in the field of place names. Because BERT's pre-training is a self-supervised Masked language model and nsp, a large amount of unlabeled data can be used for pre-training, and then the model can be fine-tuned using the corpus of classification or entity recognition in the address field to obtain the abstract features of Chinese addresses. This method further frees address extraction from corpus restrictions, and can predict non-dictionary words, further enhancing generalization capabilities.

[0009] However, BERT+BiLSTM+CRF solutions still have some problems. First, China is vast, and there are many levels of Chinese addresses (such as the ten-level addresses shown in Table 2). The total amount of Chinese place names is particularly large, and the collected corpus cannot cover all provinces, cities, and regions. In particular, the place names in remote areas are complex (such as: Yuanjiang Hani, Yi, and Dai Autonomous County, Yuxi City) and the corpus is small, resulting in the trained model having a good effect on common place names, but a poor effect on place name recognition in remote areas.

[0010] In addition, although the generalization ability of this type of solution has been improved, a large number of non-standard (abbreviated, misspelled, etc.) or even wrong place names have been extracted, making it difficult to use directly in practice. Finally, no matter which solution is used, a single place name information is generated, which may cause problems such as missing address information (such as "Guanshan Avenue, Wuhan City, Hubei Province"), redundant address information (such as "Guanggu New World T1 Office Building, diagonally opposite Wuhan Third Hospital, Guanshan Avenue, Hongshan District, Wuhan City, Hubei Province, next to Gemdale Sun City"), and entangled address information (such as "Guanggu New World, Guanshan Avenue, Hongshan District, Wuchang District, Wuhan City, Hubei Province"). Summary of the invention

[0011] 1. Technical issues to be solved

[0012] In view of the above-mentioned shortcomings of the prior art, the present invention provides an address extraction and standardization method based on pre-training, which can effectively overcome the defects of the prior art such as high cost of annotated corpus and insufficient standardization of the extracted addresses.

[0013] (II) Technical solution

[0014] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0015] A method for address extraction and standardization based on pre-training, comprising the following steps:

[0016] S1. Collect corpus containing address information and pre-train the model;

[0017] S2. Based on the enhanced address corpus, the pre-trained model is fine-tuned through a semi-supervised self-learning mode, and the fine-tuned model is used for place name recognition;

[0018] S3, performing address correction based on a self-updating and self-maintaining dictionary;

[0019] S4. Generate a model based on the multi-head attention mechanism to perform address normalization.

[0020] Preferably, in S2, based on the enhanced address corpus, the model is fine-tuned through a semi-supervised self-learning mode, including:

[0021] Obtain an address whitelist based on national administrative divisions, randomly combine addresses of all levels from the address whitelist, and construct an address whitelist corpus;

[0022] Use the addresses in the address whitelist corpus to randomly replace the corresponding level slots in the existing real corpus to construct an enhanced address corpus;

[0023] The pre-trained model is fine-tuned using the enhanced address corpus, and the fine-tuned model is used as the recognition model for place name recognition.

[0024] Preferably, the enhanced address corpus is enhanced by a semi-supervised self-learning mode, including:

[0025] Use dynamic methods to calculate the threshold for selecting corpora and extract the corpora to be annotated in batches;

[0026] After using the recognition model to identify place names, select the prediction results with higher confidence and incorporate them into the training corpus;

[0027] Perform multiple rounds of data extraction on the annotated corpus in turn, and adjust the extraction amount in each round according to the prediction results. If the prediction results are good, the extraction amount can be increased, otherwise it can be reduced.

[0028] The manually checked prediction results are fused with the original manually annotated corpus to form an enhanced address corpus.

[0029] Preferably, the pre-trained model is started by fine-tuning using a small amount of manually annotated corpus.

[0030] Preferably, address correction is performed based on a self-updating and self-maintaining dictionary in S3, including:

[0031] Split the place names obtained by the recognition model through place name recognition to obtain the place names of the previous level of the level where the place names are located and the common names of the place names;

[0032] A candidate set of place names is identified from the place names of the previous level, and rule processing is performed on the place names in the identified place names and the place names in the candidate set to separate the common place names and the specific place names;

[0033] The self-updating and self-maintaining dictionary is used to perform maximum forward matching and maximum reverse matching on place names and proper nouns, and the matching results are weighted matching, and the result with the highest weight is output as the labeled place name.

[0034] Preferably, the maximum forward matching and the maximum reverse matching of the place name and proper name include:

[0035] Perform a full match on the standard place name. If there is a full match, the match is successful and the standard place name is output. Otherwise, the matching range is expanded and the self-updating and self-maintaining dictionary is used to perform a full match on the alias place name. If there is a full match, the match is successful.

[0036] If none of the above is completely matched, rule matching is performed. If there is a rule matching result, the next step of weighted matching is entered, otherwise the matching process is directly exited.

[0037] Preferably, the weighted matching includes:

[0038] Use weighted matching based on pinyin fuzzy matching for the keywords in the rule matching results to obtain all weights.

[0039] Preferably, the self-updating and self-maintaining dictionary is extracted from actual conversation data, and a dictionary is constructed for each place name with an alias. The key of the dictionary is the corresponding standard place name, and the value of the dictionary is the corresponding alias and the number of times the alias appears. The self-updating and self-maintaining dictionary automatically updates the alias sorting according to the number of times the alias appears.

[0040] Preferably, S4 performs address normalization based on a multi-head attention mechanism generation model, including:

[0041] The corrected address data is input into the encoding network for encoding, and a multi-head attention mechanism is set between the address data and the context vector. The output vector of the encoding network is input into the decoder for decoding. The decoder generates the result and the multi-head attention vector containing the context vector generated by the multi-head attention mechanism is input into the copy network. The copy network decides whether to generate words from the vocabulary or directly copy the words in the original text, and uses the output of the copy network as the final result.

[0042] Preferably, the multi-head attention mechanism generation model includes an encoding network, a decoder and a replication network, the encoding network adopts a bidirectional long short-term memory network, the decoder adopts a unidirectional long short-term memory network, and the replication network adopts a Point network.

[0043] (III) Beneficial effects

[0044] Compared with the prior art, the address extraction and standardization method based on pre-training provided by the present invention has the following beneficial effects:

[0045] 1) In the task of place name recognition, the training corpus required for model fine-tuning is constructed by combining manual construction, hybrid methods and semi-supervised self-learning mode. This not only effectively reduces the cost of annotated corpus required for model fine-tuning, but also enhances place names with less occurrences, thereby improving the accuracy of later recognition;

[0046] 2) For the irregular addresses identified in the BERT pre-training model, the present invention performs correction matching based on a self-updating and self-maintaining dictionary, making the recognition results more effective and accurate;

[0047] 3) In response to the problems of repetition, redundancy, missing, partial hierarchical errors, etc. that often appear in address texts, the present invention generates a model based on a multi-head attention mechanism to normalize the address text, so that a standard address can be output. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a schematic diagram of the process of performing corpus enhancement through a semi-supervised self-learning mode in the present invention;

[0050] Figure 2 It is a schematic diagram of the process of performing address correction based on the self-updating and self-maintaining dictionary in the present invention;

[0051] Figure 3 The figure is a flow chart of address normalization based on the multi-head attention mechanism generation model in the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Table 1 Different address statements for the same standard address and corresponding problems

[0054]

[0055] Table 2 Ten-level address hierarchy

[0056] Label Address feature level type Example PROV Province Provinces, municipalities, autonomous regions, etc. CITY city Cities, autonomous prefectures, etc. ADNAME District / County District, county, county-level city, etc. TOWN town Town, township, etc. VIL Village / Community Village, village committee, community, etc. ROAD the way Road, avenue, national highway, etc. StreetNUM Street Number Street, Lane, etc. BuildingNUM Building No. Building, Floor, Block A, etc. LOCATION Location Point Squares, buildings, hospitals, schools, communities, etc. DIRECTION position East, West, South, North, Next, Opposite, etc.

[0057] A pre-trained address extraction and normalization method, such as Figure 1 As shown, including:

[0058] ① Collect corpus containing address information and pre-train the BERT-WWM model

[0059] ② Based on the enhanced address corpus, the pre-trained model is fine-tuned through the semi-supervised self-learning mode, and the fine-tuned model is used for place name recognition

[0060] Specifically include:

[0061] Obtain an address whitelist based on national administrative divisions, randomly combine addresses of all levels from the address whitelist, and construct the address whitelist corpus;

[0062] Use the addresses in the address whitelist corpus to randomly replace the corresponding level slots in the existing real corpus to construct an enhanced address corpus;

[0063] The pre-trained model is fine-tuned using the enhanced address corpus (the pre-trained model is first started with fine-tuning using a small amount of manually annotated corpus), and the fine-tuned model is used as a recognition model for place name recognition.

[0064] Among them, the enhanced address corpus is enhanced through a semi-supervised self-learning model, including:

[0065] Use dynamic methods to calculate the threshold for selecting corpora and extract the corpora to be annotated in batches;

[0066] After using the recognition model to identify place names, select the prediction results with higher confidence and incorporate them into the training corpus;

[0067] Perform multiple rounds of data extraction on the annotated corpus in turn, and adjust the extraction amount in each round according to the prediction results. If the prediction results are good, the extraction amount can be increased, otherwise it can be reduced.

[0068] The manually checked prediction results are fused with the original manually annotated corpus to form an enhanced address corpus.

[0069] The corpus obtained during place name recognition has more appearances in more popular areas and less appearances in less popular areas, or even no appearance at all. The corpus can be enhanced by artificial standard addresses, but the generalization ability of artificial standard addresses is too poor, and the effect in actual use is greatly reduced. In the technical solution of this application, artificial address data, annotated address data, and enhanced address corpus constructed based on enhanced data of real addresses and annotated address data are used as training corpus. The enhanced address corpus covering all addresses can improve the model recognition effect, especially for place name recognition in remote areas.

[0070] Each slot information in place name recognition can also be regarded as a classification problem. For this type of problem, self-learning can be used to enhance the corpus, which is not only highly accurate but also very versatile. Using the semi-supervised self-learning model, the annotated corpus is obtained from a large amount of corpus to be annotated as training data for model fine-tuning, which can further get rid of the dependence on annotated corpus and save the time and labor cost of corpus annotation.

[0071] ③ Address correction based on self-updating and self-maintaining dictionary

[0072] Specifically include:

[0073] Split the place names obtained by the recognition model through place name recognition to obtain the place names of the previous level of the level where the place names are located and the common names of the place names;

[0074] A candidate set of place names is identified from the place names of the previous level, and the place names in the candidate set of place names are processed according to rules to separate common place names and specific place names (common place names such as "county" and "village", and specific place names such as "Tongshan", etc.);

[0075] The self-updating and self-maintaining dictionary is used to perform maximum forward matching and maximum reverse matching on place names and proper nouns, and the matching results are weighted matching, and the result with the highest weight is output as the labeled place name.

[0076] Among them, the maximum forward matching and maximum reverse matching of place names and proper names include:

[0077] Perform a full match on the standard place name. If there is a full match, the match is successful and the standard place name is output. Otherwise, the matching range is expanded and the self-updating and self-maintaining dictionary is used to perform a full match on the alias place name. If there is a full match, the match is successful.

[0078] If none of the above is completely matched, rule matching is performed. If there is a rule matching result, the next step of weighted matching is entered, otherwise the matching process is directly exited.

[0079] Among them, weighted matching includes:

[0080] Use weighted matching based on pinyin fuzzy matching for the keywords in the rule matching results to obtain all weights.

[0081] The self-updating and self-maintaining dictionary is extracted from the actual conversation data, and a dictionary is constructed for each place name with an alias. The key of the dictionary is the corresponding standard place name, and the value of the dictionary is the corresponding alias and the number of times the alias appears. The self-updating and self-maintaining dictionary automatically updates the alias ranking according to the number of times the alias appears. On the one hand, it can improve the efficiency of later matching, and on the other hand, it can downgrade the ranking of aliases that are no longer used.

[0082] Since address planning is still changing and for some historical reasons, there is a very special phenomenon in place names, that is, there are abbreviations and nicknames for place names, for example: "Guangxi Zhuang Autonomous Prefecture" is abbreviated as "Guangxi", "Liangshan Yi Autonomous Prefecture" is abbreviated as "Liangshan", and "Yuzhou City" is also known as "Yu County".

[0083] In order to solve this type of problem, the technical solution of the present application constructs an extended dictionary of place names that can be self-updated and self-maintained. The dictionary can be automatically updated and maintained in real time based on actual conversation data, overcoming the problem of difficult dictionary maintenance during rule matching. At the same time, by matching the recognition results, the accuracy of the BERT pre-trained model in recognizing place names is effectively improved.

[0084] ④ Address normalization based on multi-head attention mechanism generation model

[0085] Specifically include:

[0086] The corrected address data is input into the encoding network for encoding, and a multi-head attention mechanism is set between the address data and the context vector (thereby increasing the encoding dimension of the address data and paying more attention to the key place name information). The output vector of the encoding network is input into the decoder for decoding. The decoder generates the result and the multi-head attention vector containing the context vector generated by the multi-head attention mechanism is input into the copy network. The copy network decides whether to generate words from the vocabulary or directly copy the words in the original text, and uses the output of the copy network as the final result.

[0087] Among them, the multi-head attention mechanism generation model includes an encoding network, a decoder and a replication network. The encoding network adopts a bidirectional long short-term memory network, the decoder adopts a unidirectional long short-term memory network, and the replication network adopts a Point network.

[0088] After the above three steps, most of the place name information is already standardized. However, the overall address text has problems such as missing elements, redundant information, disordered hierarchies, and information entanglement. It is not a standardized address. To solve this problem, the technical solution of this application is based on a multi-head attention mechanism generation model to normalize addresses. On the basis of the generation model, it integrates the multi-head attention mechanism and the replication mechanism, and obtains a standard address based on a non-standard address text by generating rather than combining.

[0089] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for address extraction and standardization based on pre-training, characterized in that: The following steps are involved: S1. Collect corpus containing address information and pre-train the model; S2. Based on the enhanced address corpus, the pre-trained model is fine-tuned through a semi-supervised self-learning mode, and the fine-tuned model is used for place name recognition; S3, performing address correction based on a self-updating and self-maintaining dictionary; S4, address normalization based on the multi-head attention mechanism generation model; S3 performs address correction based on a self-updating and self-maintaining dictionary, including: Split the place names obtained by the recognition model through place name recognition to obtain the place names of the previous level of the level where the place names are located and the common names of the place names; A candidate set of place names is identified from the place names of the previous level, and rule processing is performed on the place names in the identified place names and the place names in the candidate set to separate the common place names and the specific place names; Use the self-updating and self-maintaining dictionary to perform maximum forward matching and maximum reverse matching on place names, perform weighted matching on the matching results, and output the result with the highest weight as the labeled place name; The maximum forward matching and the maximum reverse matching of place names and proper nouns include: Perform a full match on the standard place name. If there is a full match, the match is successful and the standard place name is output. Otherwise, the matching range is expanded and the self-updating and self-maintaining dictionary is used to perform a full match on the alias place name. If there is a full match, the match is successful. If none of the above is completely matched, then rule matching is performed. If there is a rule matching result, then the next step of weighted matching is performed. Otherwise, the matching process is directly exited; The weighted matching includes: Use weighted matching based on pinyin fuzzy matching for keywords in the rule matching results to obtain all weights; Among them, the self-updating and self-maintaining dictionary is extracted from the actual conversation data, and a dictionary is constructed for each place name with an alias. The key of the dictionary is the corresponding standard place name, and the value of the dictionary is the corresponding alias and the number of times the alias appears. The self-updating and self-maintaining dictionary automatically updates the alias sorting according to the number of times the alias appears.

2. The method for address extraction and standardization based on pre-training according to claim 1, characterized in that: In S2, based on the enhanced address corpus, the model is fine-tuned through a semi-supervised self-learning mode, including: Obtain an address whitelist based on national administrative divisions, randomly combine addresses of all levels from the address whitelist, and construct an address whitelist corpus; Use the addresses in the address whitelist corpus to randomly replace the corresponding level slots in the existing real corpus to construct an enhanced address corpus; The pre-trained model is fine-tuned using the enhanced address corpus, and the fine-tuned model is used as the recognition model for place name recognition.

3. The method for address extraction and standardization based on pre-training according to claim 2, characterized in that: The enhanced address corpus is enhanced by a semi-supervised self-learning model, including: Use dynamic methods to calculate the threshold for selecting corpora and extract the corpora to be annotated in batches; After using the recognition model to identify place names, select the prediction results with higher confidence and incorporate them into the training corpus; Perform multiple rounds of data extraction on the annotated corpus in turn, and adjust the extraction amount in each round according to the prediction results. If the prediction results are good, increase the extraction amount, otherwise reduce the extraction amount; The manually checked prediction results are fused with the original manually annotated corpus to form an enhanced address corpus.

4. The method for address extraction and standardization based on pre-training according to claim 2 or 3, characterized in that: The pre-trained model is started by fine-tuning using a small amount of manually annotated corpus.

5. The method for address extraction and standardization based on pre-training according to claim 1, characterized in that: In S4, the address is normalized based on the multi-head attention mechanism generation model, including: The corrected address data is input into the encoding network for encoding, and a multi-head attention mechanism is set between the address data and the context vector. The output vector of the encoding network is input into the decoder for decoding. The decoder generates the result and the multi-head attention vector containing the context vector generated by the multi-head attention mechanism is input into the copy network. The copy network decides whether to generate words from the vocabulary or directly copy the words in the original text, and uses the output of the copy network as the final result.

6. The method for address extraction and standardization based on pre-training according to claim 5, characterized in that: The multi-head attention mechanism generation model includes an encoding network, a decoder and a replication network. The encoding network adopts a bidirectional long short-term memory network, the decoder adopts a unidirectional long short-term memory network, and the replication network adopts a Point network.

Images