A method for constructing a structured address library based on unstructured address data

Abstract

The application relates to the technical field of deep learning, and discloses a method for constructing a structured address library based on unstructured address data, which comprises the following steps: constructing an initial address tree; extracting address elements from address texts; mapping the address elements to the initial address tree; constructing and training an address element error correction model; using the address element error correction model and correcting the address elements; and pruning the initial address tree to obtain a final address tree; the address elements in unstructured address texts are extracted, the address elements are automatically corrected through a neural network model, and the unstructured address texts are constructed into a structured address tree.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, more particularly, it relates to a method for constructing a structured address library based on unstructured address data. BACKGROUND

[0002] The existing system has high requirements for the granularity of address data, but due to the complex source of address data, the address data is unstructured, and the address elements in the address data need to be identified and corrected, and a structured address tree is constructed based on unstructured address data. SUMMARY

[0003] The present application provides a method for constructing a structured address library based on unstructured address data, which solves the technical problem of identifying and correcting address elements in address data and constructing a structured address tree based on unstructured address data in related technologies.

[0004] The present application provides a method for constructing a structured address library based on unstructured address data, comprising the following steps:

[0005] Step 1: Construct an initial address tree;

[0006] The initial address tree includes 8 levels;

[0007] Step 2: Extract address elements from address text;

[0008] Extract address elements from address text, and an address element is a word;

[0009] Step 3: Map address elements to the initial address tree;

[0010] An address text generates an address element set, denoted as A={a1…a n},n<9, the order of address elements in set A is the same as the order of address elements in address text; based on named entity recognition, map address elements to the levels of the initial address tree, and establish connection between adjacent address elements in set A through a one-way pointer;

[0011] Step 4: Construct and train an address element correction model;

[0012] Generate an address chain feature matrix based on the address chain of the initial address tree;

[0013] The address chain feature matrix is represented as: Wherein represents the element in the nth column of the s row, and represents the s dimension of the mixed vector of the address element of the nth level;

[0014] The mixed vector is obtained by combining the word vector and the position vector, and the mixed vector of the address element of the nth level corresponding to the word vector, corresponding to the position vector;

[0015] If the nth address element of the address chain is empty, the value of each dimension of the mixed vector of the address element of the nth level is interpolated as 0;

[0016] The output of the address element error correction model is mapped to eight classification spaces, and the ith classification space is represented as wherein respectively represent five error types of the address element of the ith level, and the five error types are address correct, address wrong word, address level sequence disorder, address missing, and address wrong word and address level sequence disorder appearing at the same time.

[0017] Step five: using the address element error correction model and correcting the address element;

[0018] Step six: pruning the initial address tree to obtain the final address tree;

[0019] Further, each level of the initial address tree corresponds to a country, a province, a city, a district\county, a street, a community\village committee, a road\street\lane\alley\lane\natural village, and a community, respectively.

[0020] Further, the address element extraction from the address text includes the following steps:

[0021] Step 101, address text cleaning, removing useless characters and punctuation marks in the address text, such as redundant spaces and punctuation marks;

[0022] Step 102, address text segmentation, obtaining address elements by segmenting the address text through a segmentation tool;

[0023] Further, the named entity recognition is to sequentially map each element of an address element set to each level of the initial address tree.

[0024] Further, the word vector is obtained by a word vector model, and the position vector is obtained by the following calculation formula:

[0025] PE (pos,2i) =sin(pos / 10000 2i / d )

[0026] PE (pos,2i+1) =cos(pos / 10000 2i / d )

[0027] wherein pos represents the position index of the pos-th address element in the address element set, d represents the dimension of the position vector, 2i represents the even dimension of the position vector, and 2i+1 represents the odd dimension of the position vector;

[0028] Further, the address element error correction model comprises an encoder and a decoder, the encoder inputs the address chain feature matrix and outputs an encoding matrix, the address chain feature matrix and the encoding matrix have the same dimension, and the decoder inputs the encoding matrix and outputs eight connected classifiers, and the classification space of the i-th classifier is represented as wherein respectively represent five error types of the address element of the i-th layer, and the five error types are address correct, address wrong, address level sequence disorder, address missing, and address wrong and address level sequence disorder simultaneously;

[0029] Further, using the address element error correction model and correcting the address element comprises the following steps:

[0030] Step 201, inputting an address chain feature matrix of an initial address tree and outputting an error type of each address element of the address chain;

[0031] Step 202, traversing from the last address element of the address chain upwards until the error type of the traversed address element is correct, if the correct node can be traversed to, then entering step 203, otherwise entering step 204;

[0032] Step 203, traversing a complete address containing the correct address element from the complete address library;

[0033] Step 204, traversing from the last address element of the address chain upwards, traversing all address elements with the error type of address level sequence disorder as first address elements, and traversing a complete address containing the first address element from the complete address library;

[0034] Step 205, selecting one of the complete addresses obtained in step 203 or step 204 that is most matched with the current address chain as a final complete address, and correcting the current address chain based on the final complete address;

[0035] Further, the correction manner is replacing the current address chain with the address chain based on the final complete address;

[0036] Further, the matching degree of the complete address and the current address chain can be calculated by a neural network, and the classification space of the neural network is a matching degree score after discretization;

[0037] Further, the final address tree is obtained by deleting the duplicate address chains of the initial address tree.

[0038] The application has the beneficial effect that: by extracting address elements in unstructured address text, and automatically correcting address elements through a neural network model, the unstructured address text is constructed into a structured address tree. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a method step diagram for constructing a structured address library based on unstructured address data according to the application. DETAILED DESCRIPTION

[0040] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is merely meant to provide a better understanding of the subject matter described herein and can include changes, modifications, or additions of elements to the function and arrangement of the elements discussed without departing from the scope of the present disclosure. Various examples can omit, substitute, or add various procedures or components as appropriate. Also, features described with respect to some examples can be combined in other examples.

[0041] As shown in Figure 1 A method for constructing a structured address library based on unstructured address data includes the following steps:

[0042] Step one: constructing an initial address tree;

[0043] The initial address tree includes 8 levels, each level corresponds to a country, a province, a city, a district\county, a street, a community\village committee, a road\street\lane\alley\lane\natural village, and a community, respectively;

[0044] Step two: extracting address elements from address text;

[0045] An address element is a word extracted from address text;

[0046] For example, Anhui Province or Hefei City;

[0047] Extracting address elements from address text includes the following steps:

[0048] Step 101, address text cleaning, removing useless characters and punctuation marks in address text, such as redundant spaces and punctuation marks;

[0049] Step 102, address text segmentation, address text is segmented by a segmentation tool to obtain address elements;

[0050] Step three: address element mapping initial address tree;

[0051] An address text generates a set of address elements, denoted as A={a1…an}, where n is the number of address elements in the address text; n}, n < 9, the order of the address elements in set A is the same as the order of the address elements in the address text; based on named entity recognition, the address elements are mapped to the levels of the initial address tree, and a connection is established between adjacent address elements in set A through a one-way pointer;

[0052] The named entity recognition is sequentially mapping each element of an address element set to each level of the initial address tree;

[0053] For example, Dangtu County is mapped to the county level of the initial address tree;

[0054] Step four: constructing and training an address element error correction model;

[0055] Based on the initial address tree, an address chain feature matrix is generated;

[0056] The address chain feature matrix is represented as: Wherein represents the element in the nth column and the s row, and represents the s dimension of the mixed vector of the address element of the nth level;

[0057] The mixed vector is obtained by combining the word vector and the position vector, and the mixed vector of the address element of the nth level is corresponding to the dimension of the word vector, corresponding to the dimension of the position vector.

[0058] If the nth address element of the address chain is empty, the value of each dimension of the mixed vector of the address element of the nth level is interpolated as 0.

[0059] The output of the address element error correction model is mapped to eight classification spaces, and the ith classification space is represented as Wherein respectively represent the five error types of the address element of the ith level, and the five error types are address correct, address wrong word, address level order disorder, address missing, and address wrong word and address level order disorder.

[0060] In an embodiment of the present application, the word vector is obtained by a word vector model, such as a Word2vec model; the position vector is obtained by the following calculation formula:

[0061] PE (pos,2i) =sin(pos / 10000 2i / d )

[0062] PE (pos,2i+1) =cos(pos / 10000 2i / d )

[0063] Wherein, pos represents the position index of the pos-th address element in the address element set, d represents the dimension of the position vector, 2i represents the even dimension of the position vector, and 2i+1 represents the odd dimension of the position vector;

[0064] In an embodiment of the present application, the address element error correction model is a Transformer neural network model, which includes an encoder and a decoder.

[0065] The encoder inputs the address chain feature matrix and outputs an encoding matrix; the address chain feature matrix and the encoding matrix have the same dimension.

[0066] The decoder inputs the encoding matrix and outputs a softmax classification layer; the output is connected to eight classifiers, and the classification space of the i-th classifier is represented as Wherein The five error types of the i-th layer address element are represented as address correct, address wrong word, address level sequence disorder, address missing, and address wrong word and address level sequence disorder simultaneously.

[0067] The training of the encoder and the decoder is a conventional technical means, and will not be described here.

[0068] Step five: using the address element error correction model and correcting the address elements; including the following steps:

[0069] Step 201: inputting an address chain feature matrix of an initial address tree, and outputting the error type of each address element of the address chain;

[0070] Step 202: traversing from the last address element of the address chain upwards until the error type of the traversed address element is correct; if the correct node can be traversed, step 203 is entered, otherwise step 204 is entered.

[0071] Step 203: traversing the complete address containing the correct address element from the complete address library.

[0072] Step 204: traversing from the last address element of the address chain upwards, traversing all address elements with the error type of address level sequence disorder as the first address element, and traversing the complete address containing the first address element from the complete address library.

[0073] Step 205: selecting one of the complete addresses obtained in step 203 or step 204 that best matches the current address chain as the final complete address, and then correcting the current address chain based on the final complete address.

[0074] In one embodiment of the present application, the correction mode is to replace the current address chain with an address chain generated based on the final complete address.

[0075] In one embodiment of the present application, the matching degree of the complete address and the current address chain can be calculated by a neural network, and the classification space of the neural network is the matching degree score after discretization.

[0076] Step six: pruning the initial address tree to obtain a final address tree;

[0077] In one embodiment of the present application, the final address tree is obtained by deleting the duplicate address chains of the initial address tree.

[0078] The above describes the embodiments of the present embodiment, but the present embodiment is not limited to the specific embodiments described above, and the specific embodiments described above are only illustrative but not restrictive, and those skilled in the art can make many forms under the inspiration of the present embodiment, which all belong to the protection of the present embodiment.

Claims

1. A method for constructing a structured address database based on unstructured address data, characterized in that, Includes the following steps: Step 1: Construct the initial address tree; The initial address tree consists of 8 levels; Step 2: Extract address elements from the address text; Extract address elements from address text; each address element is a word. Step 3: Map address elements to the initial address tree; An address text generates a set of address elements, represented as ,gather The order of address elements in the address text is the same as the order of address elements in the address text; based on named entity recognition, address elements are mapped to the hierarchy of the initial address tree, and the set is... Adjacent address elements are connected via one-way pointers; Step 4: Build and train the address element error correction model; Generate an address chain feature matrix based on the address chain of the initial address tree; The address chain feature matrix is ​​represented as follows: ,in This represents the element in the nth column and sth row, and represents the sth dimension of the mixed vector of the address elements of the nth level; The blended vector is obtained by combining word vectors and location vectors. The blended vector of the address element at the nth level... Corresponding to the dimension of word vectors, The dimension of the corresponding position vector; If the nth address element in the address chain is empty, then the values ​​of each dimension of the blend vector of the address element at the nth level are interpolated to 0; The output of the address element error correction model is mapped to eight classification spaces, where the i-th classification space is represented as... ,in - These represent five error types for the address elements at level i: correct address, misspelled address, incorrect address hierarchy, missing address, and both misspelled address and incorrect address hierarchy. Step 5: Use the address feature error correction model to correct the address features; Using an address feature error correction model and correcting address features includes the following steps: Step 201: Input an address chain feature matrix of the initial address tree, and output the error type of each address element in the address chain; Step 202: Traverse upwards from the last address element in the address chain until the error type of the traversed address element is correct. If the correct node can be traversed, proceed to step 203; otherwise, proceed to step 204. Step 203: Traverse the complete address database containing the correct address elements; Step 204: Traverse upwards from the last address element in the address chain, taking all address elements with the error type of disordered address hierarchy as the first address element, and traverse the complete address containing the first address element from the complete address database. Step 205: Select the one that best matches the current address chain from the complete addresses obtained in step 203 or step 204 as the final complete address, and then correct the current address chain based on the final complete address; The correction method is to generate an address chain based on the final complete address and replace the current address chain; The matching degree between the complete address and the current address chain is calculated using a neural network. The classification space of the neural network is the discretized matching degree score. Step 6: Prune the initial address tree to obtain the final address tree.

2. The method for constructing a structured address database based on unstructured address data according to claim 1, characterized in that, Each level of the initial address tree corresponds to a country, province, city, district / county, street, community / village committee, road / street / lane / neighborhood / alley / natural village, and residential area.

3. The method for constructing a structured address database based on unstructured address data according to claim 1, characterized in that, Extracting address elements from address text includes the following steps: Step 101: Address text cleaning, removing useless characters and punctuation marks from the address text, including redundant spaces and punctuation marks; Step 102: Address text segmentation. The address text is segmented using a word segmentation tool to obtain address elements.

4. The method for constructing a structured address database based on unstructured address data according to claim 1, characterized in that, Named entity recognition is the process of sequentially mapping each element of a set of address features to each level of an initial address tree.

5. The method for constructing a structured address database based on unstructured address data according to claim 1, characterized in that, Word vectors are obtained through a word vector model, and position vectors are obtained using the following formula: ； ； Where pos represents the position index of the pos-th address element in the address element set, and d represents the dimension of the position vector. The even-numbered dimension of the position vector. This represents the odd-numbered dimension of the position vector.

6. The method for constructing a structured address database based on unstructured address data according to claim 1, characterized in that, The address feature error correction model includes an encoder and a decoder. The encoder takes the address chain feature matrix as input and outputs an encoding matrix; the address chain feature matrix and the encoding matrix have the same dimension. The decoder takes the encoding matrix as input and outputs a matrix connected to eight classifiers. The classification space of the i-th classifier is represented as... ,in - These represent five error types for the address elements at level i: correct address, misspelled address, incorrect address hierarchy, missing address, and both misspelled and incorrect address hierarchy.

7. The method for constructing a structured address database based on unstructured address data according to claim 1, characterized in that, The final address tree is obtained by removing duplicate address chains from the initial address tree.

Images