A method for extracting city, administrative division and country based on English address

Abstract

The application relates to the technical field of natural language processing, and particularly discloses a method for extracting cities, administrative divisions and countries based on English address, which comprises the following steps: inputting English address text data, using a pre-trained BERT model to fuse LSTM to perform binary classification of institutions and addresses, performing data preprocessing on the address data, performing data analysis on the processed address data, matching according to the analysis result, outputting the matching result according to the level after the matching is completed, and using the method of pre-processing and matching the regional level dictionary and the standard document through the rule to solve the problem that the cities, administrative divisions and country texts of the English address text data of the scientific research literature cannot be accurately extracted when the rule is matched, the level is disordered and noise exists; and the problem that the cost of manually constructing a training data set is high when a machine learning model is used is also solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a city, administrative division and country recognition extraction method based on English address. BACKGROUND

[0002] With the development of science and technology, the number of published scientific research literatures in various fields is increasing day by day, and the accumulated scientific research literatures in various fields and disciplines in the past are more and more difficult to classify, manage and analyze. With the development of information technology, more and more scientific research institutions hope to use computers to automatically classify, manage and analyze various scientific research literatures. Through accurate recognition and extraction of city, administrative division and country texts in English address texts composed of organization (or organization abbreviation), city, administrative division, postcode and country information in scientific research literatures, the city, administrative division and country texts can be used to assist in classifying, analyzing and managing the scientific research literatures according to the city, administrative division and country, and can also be used to assist in standardizing the organization information of the scientific research literatures.

[0003] There are mainly two kinds of existing address information extraction technologies: one is a rule matching based method, which uses rules to match address texts to extract corresponding information by summarizing address hierarchy distribution rules; and the other is a machine learning model based method, which trains a machine learning model based on a large amount of manually annotated address text data, inputs address text data to be extracted into the trained machine learning model, and extracts corresponding information through the machine learning model.

[0004] The above two kinds of address information extraction technologies still have the following defects respectively:

[0005] 1. The rule matching based method excessively depends on preset information extraction rules, and cannot accurately extract English address text data of scientific research literatures when the hierarchy order of the address text data is changed or there are too many interference items (such as organization, postcode, mailbox and special symbols) in the address text data;

[0006] 2. The machine learning model based method needs a large amount of annotated English address text data of scientific research literatures to train the model, and these annotated English address text data of scientific research literatures need to be extracted from a large amount of scientific research literatures and manually annotated, which requires a long time and high cost to manually establish a training data set.

[0007] Therefore, we need to propose a city, administrative division and country recognition extraction method based on English address to solve the above problems. SUMMARY

[0008] The application aims to provide an extraction method for city, administrative division and country identification based on English address, which uses a pre-trained BERT model to fuse LSTM and a method of matching regional hierarchical dictionaries and standard documents through rules, solves the problem of accurately extracting city, administrative division and country text of scientific research literature English address text data with hierarchical disorder and noise based on rule matching, and solves the problems in the background art.

[0009] To achieve the above-mentioned purpose, the application provides the following technical scheme: an extraction method for city, administrative division and country identification based on English address, comprising the following steps:

[0010] S1, inputting English address text and using a pre-trained BERT model to fuse LSTM through an institutional address two-classification module to divide the input data into two-class data of institutional data and address data;

[0011] S2, performing data preprocessing of cleaning and sorting on the English address text input data;

[0012] S3, performing data analysis on the address segmentation data after data preprocessing, and judging whether there is corresponding information of city, administrative division and country;

[0013] S4, performing data matching according to the data analysis result, matching the corresponding standard documents of city, administrative division and country, and finally outputting the matched standard information of city, administrative division and country according to the hierarchy.

[0014] Preferably, the institutional address two-classification module receives an original address string as input, divides the original address string through an address separator, traverses the division result, judges whether the current traversal element is a country or a city, and if yes, skips; if not, calls a model to judge whether the current element is an institution, and if yes, records, if not, skips, finally, cuts out the address information through the recorded institutional information, and returns the institutional and address information.

[0015] Preferably, the institutional address two-classification module comprises the following steps when performing segmentation:

[0016] A1, inputting original address information;

[0017] A2, dividing the original address;

[0018] A3, traversing the divided address and confirming whether the traversal is ended, which has the following two situations:

[0019] A31, when the traversal is ended, dividing the original address into institutional and address data according to the institutional information, and then outputting the institutional and address information;

[0020] A32, When the traversal is not finished, confirm whether the current element is a country or a city, match the country and city standard document, there are two cases:

[0021] A321, If the current element is a country or a city, return to A2;

[0022] A322, If the current element is not a country or a city, call the model again, and confirm whether the current element is an organization, there are two cases:

[0023] A3221, If the current element is not an organization, return to A2;

[0024] A3222, If the current element is an organization, record the organization information.

[0025] Preferably, in step S2, when the address data is pre-processed, since the input data has noise, the noise refers to data invalid for extraction and interfering with the extraction result, so the data is cleaned and sorted before data analysis, that is, the useless information at both ends of the input address information is removed, the noise data of non-address information is cleaned, the address is divided into multiple blocks by an address separator, and the division result is sorted in reverse order.

[0026] Preferably, the data pre-processing process is as follows:

[0027] B1, after the address data is input, confirm whether the address data is empty, there are two cases:

[0028] B11, when the input address data is empty, the address data input is performed again, and B1 is performed;

[0029] B12, if the address data is not empty, the address data is cleaned, and B2 is performed;

[0030] B2, divide the address information, and confirm whether the division result is empty, there are two cases:

[0031] B21, when the division result is empty, B1 is entered;

[0032] B22, when the division result is not empty, the division result is sorted in reverse order, and B3 is entered;

[0033] B3, output the division and sorting result.

[0034] Preferably, the data is parsed in step S3 to match the established address hierarchy dictionary, specification document, and determine whether the address data exists in the city, administrative division, and country information according to the address segmentation data in descending order. If it exists, the location index is marked. Since the city, administrative division, and country data need to be continuously determined, the number of determination results is 23, that is, there are eight address hierarchy types.

[0035] Preferably, the eight address hierarchy types are not found in the city, administrative division, and country, only found in the city, only found in the administrative division, only found in the country, found in the city and administrative division, found in the city and country, found in the administrative division and country, and found in the city, administrative division, and country.

[0036] Preferably, the data parsing includes the following steps:

[0037] C1, address segmentation sorting data input, determine whether the current element is a country, there are two cases:

[0038] C11, when the current element is a country, mark the country found;

[0039] C12, when the current element is not a country, determine whether the current element is an administrative division, there are two cases:

[0040] C121, when the current element is an administrative division, determine whether the adjacent element is an administrative division, when the adjacent element is not an administrative division, mark the administrative division found;

[0041] C122, when the current element is not an administrative division or the adjacent element in C121 is an administrative division, determine whether the current element is a city, there are two cases:

[0042] C1221, when the current element is a city, determine whether the adjacent element is a city, if the adjacent element is not a city, mark the city found, and output the mark information;

[0043] C1222, when the current element is not a city or the adjacent element in C1221 is a city, return to C1.

[0044] Preferably, in step S4, the data matching is performed on the eight address hierarchy types for different processing, and finally the city, administrative division, and country specification information is output.

[0045] Preferably, the different processing of the eight address hierarchy types is as follows:

[0046] City, administrative division, and country are not found: set the city, administrative division, and country specification information to be empty and return;

[0047] Only find city: find its corresponding administrative division and country according to the city matching level dictionary, and match the standard document respectively, return the standard information;

[0048] Only find administrative division: find its corresponding country according to the administrative division matching level dictionary, and match the standard document respectively, return the standard information;

[0049] Only find country: match the standard document, return the corresponding standard information of the country;

[0050] Find city, administrative division: find its corresponding administrative division according to the city, country matching level dictionary, and match the standard document respectively, return the standard work and rest;

[0051] Find administrative division, country: match the standard document respectively, return the standard information;

[0052] City, administrative division, country have been found: city, administrative division, country match the standard document respectively, return the standard information.

[0053] Compared with the prior art, the beneficial effects of the present application are:

[0054] After the English address text data is input, the pre-trained BERT model is used to fuse LSTM to perform institution and address binary classification, then the address data is preprocessed, the processed address data is parsed, and finally the matching result is matched according to the parsing result. After the matching is completed, the matching result is output according to the level. Through the method of using preprocessing and matching the region level dictionary and the standard document by rule, the problem that the city, administrative division and country text of the scientific research literature English address text data cannot be accurately extracted when the rule is matched is solved. And the problem of high cost of manual construction of training data set when using machine learning model is also solved. Compared with other methods, the city, administrative division and country text in the scientific research literature English address text are more accurately identified and extracted, and the hierarchical relationship is maintained, which is used to assist the classification management and analysis of various scientific research literatures. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The flowchart of the present application;

[0056] Figure 2 The flowchart of the present application;

[0057] Figure 3 The flowchart of the institution address binary classification module of the present application;

[0058] Figure 4 The flowchart of the address data preprocessing of the present application;

[0059] Figure 5 Address data parsing flowchart of the present application;

[0060] Figure 6 Eight path results of address level type of the present application;

[0061] Figure 7 Address data matching flowchart of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0063] Please refer to Figures 1-6 The present application provides a technical solution: a method for extracting city, administrative division and country identification based on English address, comprising the following steps:

[0064] S1, after inputting the English address text, the institutional address two-classification module uses a pre-trained BERT model to fuse LSTM to divide the input data into two-class data of institutional data and address data. After classification, the institutional data will be excluded.

[0065] The institutional address two-classification module receives the original address string as input, divides the original address string through the address separator, traverses the division result, judges whether the current traversal element is a country or a city, and if so, skips; if not, calls the model to judge whether the current element is an institution, and if so, records it, and if not, skips; finally, the address information is cut out through the recorded institutional information, and the institutional and address information is returned.

[0066] The institutional address two-classification module includes the following steps when dividing:

[0067] A1, input the original address information;

[0068] A2, divide the original address;

[0069] A3, traverse the divided address and confirm whether the traversal is ended, which has the following two situations:

[0070] A31, when the traversal is ended, the original address is divided into institutional and address data according to the institutional information, and then the institutional and address information is outputted;

[0071] A32, When the traversal is not finished, confirm whether the current element is a country or a city, match the country and city specification document, and there are two cases:

[0072] A321, If the current element is a country or a city, return to A2;

[0073] A322, If the current element is not a country or a city, call the model again, and confirm whether the current element is an organization, and there are two cases:

[0074] A3221, If the current element is not an organization, return to A2;

[0075] A3222, If the current element is an organization, record the organization information.

[0076] S2, data preprocessing of cleaning and sorting the English address text input data;

[0077] When the address data is preprocessed, due to the existence of noise in the input data, the noise refers to the data that is invalid for extraction and will interfere with the extraction result, so the data is cleaned and sorted before data analysis, that is, the useless information at both ends of the input address information is removed, the noise data of non-address information is cleaned, and the address is divided into multiple blocks by the address separator, and the division result is sorted in reverse order;

[0078] The flow of data preprocessing is as follows:

[0079] B1, after the address data is input, confirm whether the address data is empty, and there are two cases:

[0080] B11, when the input address data is empty, re-input the address data and perform B1;

[0081] B12, if the address data is not empty, clean the address data and perform B2;

[0082] B2, divide the address information and confirm whether the division result is empty, and there are two cases:

[0083] B21, when the division result is empty, enter B1;

[0084] B22, when the division result is not empty, sort the division result in reverse order and enter B3;

[0085] B3, output the division and sorting result.

[0086] S3, data analysis of the address division data after data preprocessing, judge whether there is city, administrative division and country information;

[0087] When the data is parsed, the established address hierarchy dictionary and the specification document are matched, and whether the address data exists city, administrative division, and country information is judged according to the address segmentation data in descending order. If it exists, the location index mark is marked. Because it is necessary to continuously judge whether city, administrative division, and country data exist, the number of judgment results is 23, that is, there are eight address hierarchy types;

[0088] The eight address hierarchy types are city, administrative division, and country, none of which are found, only city is found, only administrative division is found, only country is found, city and administrative division are found, city and country are found, administrative division and country are found, and city, administrative division, and country are all found;

[0089] When the data is parsed, the following steps are included:

[0090] C1, address segmentation sorting data input, judge whether the current element is a country, there are the following two cases:

[0091] C11, when the current element is a country, mark that the country has been found;

[0092] C12, when the current element is not a country, confirm whether the current element is an administrative division, there are the following two cases:

[0093] C121, when the current element is an administrative division, judge whether the adjacent element is an administrative division, when the adjacent element is not an administrative division, mark that the administrative division has been found;

[0094] C122, when the current element is not an administrative division or the adjacent element in C121 is an administrative division, judge whether the current element is a city, there are the following two cases:

[0095] C1221, when the current element is a city, judge whether the adjacent element is a city, if the adjacent element is not a city, mark that the city has been found, and output the mark information;

[0096] C1222, when the current element is not a city or the adjacent element in C1221 is a city, return to C1.

[0097] S4, according to the data parsing result, the data is matched, the city, administrative division, and country specification document are matched, and finally the matched city, administrative division, and country specification information are output according to the hierarchy.

[0098] When the data is matched, the eight address hierarchy types are processed differently, and finally the city, administrative division, and country specification information are output;

[0099] The different processing of the eight address hierarchy types is as follows:

[0100] City, administrative division, country are not found: Set the city, administrative division, country specification information to empty and return;

[0101] Only city is found: According to the city matching level dictionary, find out its corresponding administrative division and country, and match the specification document respectively, and return the specification information;

[0102] Only administrative division is found: According to the administrative division matching level dictionary, find out its corresponding country, and match the specification document respectively, and return the specification information;

[0103] Only country is found: Match the specification document, and return the specification information of the country;

[0104] City and administrative division are found: According to the city and country matching level dictionary, find out its corresponding administrative division, and match the specification document respectively, and return the specification information;

[0105] Administrative division and country are found: The administrative division and country match the specification document respectively, and return the specification information;

[0106] City, administrative division, and country are found: City, administrative division, and country match the specification document respectively, and return the specification information.

[0107] In summary, after inputting the English address text data, using the pre-trained BERT model combined with LSTM to perform institution and address binary classification, then preprocessing the address data, processing the address data, and finally matching according to the analysis result, the matching result is output according to the level after the matching is completed. By using the preprocessing and the method of matching the region level dictionary and the specification document through the rule, the problem of accurately extracting the city, administrative division, and country text of the scientific research literature English address text data with disordered levels and noise based on rule matching is solved. And it also solves the problem of high cost of manually constructing training data set using machine learning model. Compared with other methods, it focuses more on accurate identification and extraction of city, administrative division, and country text in scientific research literature English address text, and maintains its hierarchical relationship, which is used to assist the classification management and analysis of various scientific research literature.

[0108] Although the embodiments of the present application have been shown and described, it can be understood by those skilled in the art that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for extracting city, administrative division and country identification based on English address, characterized in that: The method comprises the following steps: S1, inputting English address text, and using a pre-trained BERT model to fuse LSTM to divide the input data into two types of data of an organization and an address by an organization address two-classification module; The organization address two-classification module receives an original address string as input, divides the original address string by an address separator, traverses the division result, judges whether the current traversed element is a country or a city, and if yes, skips; if not, calls a model to judge whether the current element is an organization, and if yes, records, and if not, skips, finally, cuts out the address information by the recorded organization information, and returns the organization and address information; The organization address two-classification module comprises the following steps when dividing: A1, inputting original address information; A2, dividing the original address; A3, traversing the divided address, and confirming whether the traversal is ended, which exists the following two situations: A31, when the traversal is ended, dividing the original address into organization and address data according to the organization information, and then outputting the organization and address information; A32, when the traversal is not ended, confirming whether the current element is a country or a city, matching the country and city standard documents, which exists the following two situations: A321, if the current element is a country or a city, returning to A2; A322, if the current element is not a country or a city, calling a model again, and confirming whether the current element is an organization, which exists the following two situations: A3221, if the current element is not an organization, returning to A2; A3222, if the current element is an organization, recording the organization information; S2, data preprocessing of cleaning and sorting the address data; S3, data analysis of the address data after data preprocessing, judging whether there is corresponding information of a city, an administrative division and a country; S4, data matching according to the data analysis result, matching the city, the administrative division and the country standard documents, and finally outputting the matched city, the administrative division and the country standard information according to the level.

2. The method of claim 1, wherein the method comprises: In step S2, since the input data has noise, which means invalid data that will interfere with the extraction result, the data is cleaned and sorted before data analysis, that is, the useless information at both ends of the input address information is removed, the noise data of non-address information is cleaned, the address is divided into multiple blocks by an address separator, and the division result is sorted in reverse order.

3. The method of claim 2, wherein the method comprises: The data preprocessing process is as follows: B1, after inputting the address data, confirming whether the address data is empty, which exists the following two situations: B11, when the input address data is empty, re-inputting the address data, and performing B1; B12, if the address data is not empty, cleaning the address data, and performing B2; B2, dividing the address information, and confirming whether the division result is empty, which exists the following two situations: B21, when the division result is empty, entering B1; B22, when the division result is not empty, sorting the division result in reverse order, and entering B3; B3, outputting the sorted division result.

4. The method of claim 3, wherein the method comprises: When the data is parsed in step S3, the established address hierarchy dictionary and the specification document are matched, and it is judged whether the address data exists city, administrative division, and country information according to the address segmentation data after the reverse sorting, and if it exists, the location index mark is marked. Because it needs to continuously judge whether the city, administrative division, and country data exist, the number of judgment results is 2 3 That is, there are eight address hierarchy types.

5. The method of claim 4, wherein the method further comprises: The eight address level types are city, administrative division, and country, none of which are found, only city, only administrative division, only country, city and administrative division, city and country, administrative division and country, and city, administrative division, and country are all found.

6. The method of claim 5, wherein the method further comprises: The data analysis includes the following steps: C1, address segmentation and sorting data input, determine whether the current element is a country, there are two cases: C11, when the current element is a country, mark the country found; C12, when the current element is not a country, determine whether the current element is an administrative division, there are two cases: C121, when the current element is an administrative division, determine whether the adjacent element is an administrative division, when the adjacent element is not an administrative division, mark the administrative division found; C122, when the current element is not an administrative division or the adjacent element in C121 is an administrative division, determine whether the current element is a city, there are two cases: C1221, when the current element is a city, determine whether the adjacent element is a city, if the adjacent element is not a city, mark the city found, and output the mark information; C1222, when the current element is not a city or the adjacent element in C1221 is a city, return to C1.

7. The method of claim 6, wherein the method further comprises: In step S4, different processing is performed on the eight address level types during data matching, and finally the city, administrative division, and country specification information is output.

8. The method of claim 7, wherein the method further comprises: The different processing of the eight address level types is as follows: City, administrative division, and country are not found: set the city, administrative division, and country specification information to empty and return; Only city is found: find the corresponding administrative division and country according to the city matching level dictionary, match the specification documents respectively, and return the specification information; Only administrative division is found: find the corresponding country according to the administrative division matching level dictionary, match the specification documents respectively, and return the specification information; Only country is found: match the specification document and return the corresponding specification information of the country; City and administrative division are found: find the corresponding administrative division according to the city and country matching level dictionary, match the specification documents respectively, and return the specification information; Administrative division and country are found: match the specification documents of administrative division and country respectively, and return the specification information; City, administrative division, and country are all found: match the specification documents of city, administrative division, and country respectively, and return the specification information.

Images