A multi-source heterogeneous POI data rapid deduplication method

Abstract

The application discloses a kind of multi-source heterogeneous POI data fast deduplication method, it is related to LBS location service technical field, including data preprocessing, grid index and nine palace grid three big modules: wherein data preprocessing includes the following steps: for POI name, it is divided into words, removes stop words, statistics word frequency, obtains word frequency vector, and then calculates the word frequency in entire database;The application adopts two kinds of methods of edit distance and "low-frequency word" statistics to process non-same POI.In the past patent, literature, the method of POI deduplication almost does not involve "low-frequency word" statistics method.The probability that two different POIs simultaneously have same "low-frequency word" is very low, when the number of "low-frequency word" that they have is more than one, and straight line distance is very close, it can be considered that the two POIs are the same to a greater extent.And "low-frequency word" statistics method principle is simple, easy to operate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of LBS location service, and particularly relates to a multi-source heterogeneous POI data rapid deduplication method. BACKGROUND

[0002] With the continuous development of information technology, the sources of location information are more extensive, and data updates are more frequent. Reasonable use of location information is of great significance to related analysis and decision-making. POI data, as an important carrier of location information, is directly related to the quality of location information and is an important reference for location information research. For multi-source heterogeneous POI data, how to efficiently fuse them has become one of the problems that the industry urgently needs to solve.

[0003] At present, the POI data fusion method mainly includes methods based on spatial location, non-spatial attribute, ontology, and the combination of spatial location and non-spatial attribute. Among them, the four classical POI fusion algorithms are: POI fusion technology based on one-sided nearest neighbor connection algorithm, POI fusion method based on weighted multi-attribute similarity, POI fusion technology based on distance category, and POI fusion technology based on clustering and indexing.

[0004] The POI fusion technology based on one-sided nearest neighbor connection algorithm finds the correct fusion set through the spatial location of the object. Although this algorithm is simple to operate, it only considers the spatial location and does not consider the non-spatial attribute, resulting in inaccurate fusion results.

[0005] The POI fusion method based on weighted multi-attribute similarity assigns different weights to the name similarity, distance similarity, and address similarity, respectively. It calculates the overall similarity and judges whether the overall similarity is greater than a certain threshold to determine whether two POIs are the same. This method is simple in theory and easy to operate, but it faces an important problem: how to determine the weight of different attributes? Artificially determining the weight has strong subjective factors, which can lead to low accuracy of the data results after fusion.

[0006] The POI fusion method based on distance category mainly includes three stages: preliminary screening stage, exclusion stage, and supplementary stage. The preliminary screening stage uses the mutual nearest neighbor algorithm, and the exclusion stage and the supplementary stage use the Jaro-Winkler algorithm. These algorithms are difficult to operate and have high time complexity, and have low applicability.

[0007] POI fusion technology based on clustering and indexing aggregates POIs through clustering. This method is suitable for offline computation, but it is difficult to modify the original POI clusters in real time. The cost of maintaining the database in real time is very high. In addition, updating new data requires all data to be injected and all calculations to be completed before the new clustering results can be returned. The entity link relationships of new POIs cannot be calculated in real time. The comparison candidate set is narrowed by building an index, but this method has the problem of "index granularity". That is, if the search granularity is set too small, two POIs that should be linked by entities cannot be linked together, which will reduce the recall rate. Conversely, the returned suspected candidate set is too large, and the time to calculate the POIs linked by entities for each POI will become very long, which will increase the pressure on the engineering implementation. It is difficult to control the index granularity through character indexing.

[0008] Therefore, this invention provides a fast deduplication method for multi-source heterogeneous POI data. Summary of the Invention

[0009] In view of the aforementioned existing problems, the present invention is proposed.

[0010] Therefore, the purpose of this invention is to provide a fast deduplication method for multi-source heterogeneous POI data, which solves the problem of LBS location services.

[0011] To achieve the above objectives, the present invention provides the following technical solution:

[0012] A fast deduplication method for multi-source heterogeneous POI data includes three main modules: data preprocessing, raster indexing, and a nine-square grid. Data preprocessing includes the following steps:

[0013] For each POI name, word segmentation and stop word removal are performed, word frequency is counted to obtain a word frequency vector, and then the word frequency of the entire database is calculated.

[0014] For POI coordinates, commonly used POI coordinate systems include GCJ02, WGS84, and BD09. These should be uniformly converted to the GCJ02 coordinate system.

[0015] Furthermore, the raster index mainly includes the following:

[0016] Raster indexing, to reduce computational load, divides the geographic coordinate system into rectangular grids and establishes an inverse index between the grid and the POI coordinates.

[0017] The raster index is calculated using the following formula:

[0018]

[0019]

[0020] Among them, lgi longitude of POI i, lt i latitude of POI i, px, py are the grid line densities in east-west and north-south directions, respectively, finally, xn i , yn i grid index coordinates of POI i, and is a floor function.

[0021] Further, the nine-square module includes two small modules, which are the de-duplication of "same name" POI data in the nine-square, the de-duplication of "different name" POI data in the nine-square, and the merging of POI data. The nine-square module includes the following contents:

[0022] The nine-square, that is, the selected center grid i (xn i ,yn i ), for any other grid j (xn j ,yn j ), if the absolute value distance between grids i and j is less than or equal to 2, the center grid i and all grids j that meet the condition are called a "nine-square",

[0023] {i,j||xn i -xn j |+|yn i -yn j |≤2}

[0024] A "nine-square" contains 3x3 nine grids adjacent to any one grid in the upper, lower, left, right, upper left, lower left, upper right, and lower right directions,

[0025] If there is at least one mirror POI for a POI, the geographic coordinates of the mirror POI and the original POI coordinates are certainly less than the grid line density, that is, they must fall within the "nine-square" centered on the original POI, so a nine-square is a basic de-duplication unit.

[0026] Further, the de-duplication of "same name" POI data in the nine-square mainly includes the following contents:

[0027] For "same name" POI in the nine-square and straight-line distance less than 300m, it is determined to be the same POI and is merged, that is, when the names of two POI data are completely the same and the distance is close enough, it can be considered that the two POIs are the same, and the case of completely the same name is mainly divided into three types:

[0028] The first type: when the names of the two POIs are both Chinese-English combinations, the Chinese part and the English part correspond to the same respectively.

[0029] The second type: When two POI names, one with only Chinese names and the other with a combination of Chinese and English, the corresponding Chinese parts are the same;

[0030] The third type: When one of the two POI names is only an English name and the other is a combination of Chinese and English, the corresponding English parts are the same.

[0031] Furthermore, the deduplication of "different name" POI data within the nine-square grid is mainly achieved using two methods: edit distance and low-frequency word frequency statistics after word segmentation. This includes the following:

[0032] S1, Edit Distance

[0033] Edit distance is a quantitative measure of the difference between two strings. It measures the minimum number of operations required to transform one string into the other. Let the two strings be i and j, and their lengths be n. i and n j The edit distance between them is e i,j The similarity is sim i,j Based on the meaning of edit distance, the text similarity formula can be defined as follows:

[0034]

[0035] Where, n = max{n i ,n j Based on this, the names are further divided into two categories: those with inclusion relationships and those with other relationships.

[0036] S2. When names have an inclusion relationship, this relationship can be further divided into "sequential inclusion" and "random inclusion". Let sim be the name similarity between two POIs. i,j The straight-line distance is d i,j The number of identical Chinese characters is n i,j The judgment rules are as follows:

[0037] When two POIs have names that are "sequentially contained", they can be considered to be the same POI if the name similarity is greater than or equal to 0.8 and the straight-line distance is less than or equal to 250 meters; or the straight-line distance between the two POIs is less than 30 meters and the number of identical Chinese characters is greater than or equal to 2; or the straight-line distance between the two POIs is less than 50 meters and the number of identical Chinese characters is greater than or equal to 3.

[0038] When the names of two POIs are "out of order containing", if the name similarity is greater than or equal to 0.8 and the straight-line distance is less than or equal to 150 meters; or the name similarity is greater than or equal to 0.7 and the straight-line distance is less than or equal to 100 meters; or the straight-line distance between the two POIs is less than 10 meters and the number of same Chinese characters is greater than or equal to 4, they are considered to be the same POI.

[0039] S3, when other conditions, considering the complexity of the collected POI data, the names of two POIs have other various conditions in addition to the "same name" and "containing" relationship, and the two POIs can be determined to be the same. The most common ones mainly have three types:

[0040] The first type is that there are more than 3 consecutive numbers in the beginning and the straight-line distance is less than or equal to 100 meters;

[0041] The second type is that there are more than 4 consecutive English letters in the beginning and the straight-line distance is less than or equal to 150 meters;

[0042] The third type is to consider the homophonic characters. When the Chinese pinyin of the names of two POIs is completely the same, they are also considered to be the same POI.

[0043] S4, in the low-frequency word statistics, using the word segmentation data in the preprocessing process, when the frequency is less than 75, it is considered to be a "low-frequency word". When two POIs have one same "low-frequency word" and the straight-line distance is less than 500 meters, they are further investigated, mainly including the following four cases:

[0044] The number of same "low-frequency words" is greater than or equal to 2, the straight-line distance is less than or equal to 100 meters, the first two characters are the same, and the straight-line distance is less than or equal to 100 meters;

[0045] There is a containing relationship, and the straight-line distance is less than or equal to 100 meters;

[0046] The key words in the address except "street", "road" and "number" are highly similar, and the straight-line distance is less than or equal to 500 meters;

[0047] When one of the above conditions is met, the two POIs are determined to be the same.

[0048] Further, the POI data merging step mainly includes the following contents:

[0049] The principle of minimum matching is adopted for fusion: if a POI is identified as a duplicate POI by two methods at the same time, the POI is a duplicate POI and is removed; if a POI is identified as a duplicate POI by at most one method, the POI is not a duplicate POI and is retained. Then, the union set of the data sets obtained by removing the same name and different name is taken to obtain the final result after removing the duplicates.

[0050] In the above technical solution, the present application provides technical effects and advantages:

[0051] 1、 The present application adopts two methods of edit distance and "low-frequency word" statistics to process non-same POI. In the previous patents and documents, the method of POI deduplication almost does not involve the method of "low-frequency word" statistics. The probability of two different POIs having the same "low-frequency word" is very low, when they have more than one "low-frequency word" and the straight-line distance is very close, it can be considered that the two POIs are the same to a large extent. And the method of "low-frequency word" statistics is simple in principle and easy to operate.

[0052] 2、 The present application adopts three indexes of name similarity, straight-line distance and address similarity to comprehensively judge, compares the three indexes with certain standards, and formulates rules according to the constraints between the three indexes, instead of artificially setting weights to specify rules in the previous scheme, which reduces the influence of subjective factors to a certain extent and improves the accuracy of the deduplication result. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only illustrate some embodiments of the present application, and other drawings can also be obtained by those skilled in the art according to these drawings.

[0054] Fig. 1 It is a schematic diagram of the overall flow structure of the present application.

[0055] Fig. 2 It is a nine-square diagram of the present application.

[0056] Fig. 3 It is a schematic diagram of the repeated judgment rule of the present application. DETAILED DESCRIPTION

[0057] In order to make those skilled in the art better understand the technical solutions of the present application, the present application will be further described in detail with reference to the drawings.

[0058] The embodiment of the present application discloses a multi-source heterogeneous POI data fast deduplication method.

[0059] The present application provides a multi-source heterogeneous POI data fast deduplication method as shown in Figs. 1-3 The present application provides a multi-source heterogeneous POI data fast deduplication method as shown in

[0060] For POI name, it is segmented, stop words are removed, word frequency is counted, word frequency vector is obtained, and then word frequency in the whole database is calculated;

[0061] Example: Athletes' Physical Training Center

[0062] Fitness Center for Children

[0063] Word segmentation yielded: Athlete's Physical Training Center

[0064] Fitness Center for Children

[0065] The corresponding word frequency statistics are as follows:

[0066] Athletes: 2; Physical Fitness: 2; Training: 1; Activities: 1; Centers: 2

[0067] For POI coordinates, commonly used POI coordinate systems include GCJ02, WGS84, and BD09. These should be uniformly converted to the GCJ02 coordinate system.

[0068] The raster index mainly includes the following:

[0069] Raster indexing, to reduce computational load, divides the geographic coordinate system into rectangular grids and establishes an inverse index between the grid and the POI coordinates.

[0070] The raster index is calculated using the following formula:

[0071]

[0072]

[0073] Among them, lg i Let lt be the longitude of i in POI. i Let be the dimension of 'i' in the POI, and px and py be the raster line densities in the east-west and north-south directions, respectively. Finally, xn i yn i Here are the raster index coordinates of i in the POI. as well as This is the floor function.

[0074] The nine-grid module comprises two sub-modules: deduplication of POI data with the same name within the nine-grid, deduplication of POI data with different names within the nine-grid, and merging of POI data. The nine-grid module includes the following:

[0075] A 3x3 grid, that is, selecting the center grid i(xn) i ,yn i For any other raster j(xn) j ,yn j If the absolute distance between grid cells i and j is less than or equal to 2, then the grid consisting of the central grid cell i and all grid cells j that satisfy the condition is called a "nine-square grid".

[0076] {i,j||xn i -xn j |+|yn i -yn j |≤2}

[0077] A "nine-square" contains a center grid and its adjacent upper, lower, left, right, upper left, lower left, upper right and lower right 3x3 grids,

[0078] If the POI has at least one mirror POI, the geographic coordinates of the mirror POI and the original POI coordinates are a certain distance less than the grid line density, that is, a certain fall within the "nine-square" centered on the original POI, so a nine-square is a basic deduplication unit.

[0079] The deduplication of the "same name" POI data in the nine-square mainly includes the following contents:

[0080] For the "same name" POI in the nine-square and the straight line distance less than 300m, it is determined as the same POI and is merged, that is, when the names of two POI data are completely the same and the distance is close enough, it can be considered that the two POIs are the same, and the completely same name is mainly divided into three cases:

[0081] The first kind: when the names of the two POIs are both Chinese-English combinations, the Chinese part and the English part correspond to the same;

[0082] The second kind: when one of the two POI names has only a Chinese name and the other is a Chinese-English combination, the Chinese part corresponds to the same;

[0083] The third kind: when one of the two POI names has only an English name and the other is a Chinese-English combination, the English part corresponds to the same.

[0084] The deduplication of the "different name" POI data in the nine-square is mainly for different name data, mainly using the edit distance and the low-frequency word frequency statistics after segmentation, and mainly includes the following contents:

[0085] S1, edit distance

[0086] The edit distance is a quantitative measure of the difference between two strings, and the measurement method is to see how many times of processing is needed to change one string into another. The lengths of the two strings are n i and n j , and the edit distance between them is e i,j , and the similarity is sim i,j According to the meaning of the edit distance, the text similarity formula can be defined as follows:

[0087]

[0088] Wherein, n = max{n i ,n j}, on this basis, the name is divided into two categories of containing relationship and other cases;

[0089] S2, when the name has a containing relationship, the name with containing relationship can be divided into "sequential containing" and "disordered containing", and the name similarity between two POIs is sim i,j , the straight-line distance is d i,j , and the number of same Chinese characters is n i,j , and the determination rule is as follows:

[0090] When the names of two POIs are "sequential containing", if the name similarity is greater than or equal to 0.8 and the straight-line distance is less than or equal to 250 meters; or the straight-line distance between two POIs is less than 30 meters and the number of same Chinese characters is greater than or equal to 2; or the straight-line distance between two POIs is less than 50 meters and the number of same Chinese characters is greater than or equal to 3, it is considered that they are the same POI;

[0091] When the names of two POIs are "disordered containing", if the name similarity is greater than or equal to 0.8 and the straight-line distance is less than or equal to 150 meters; or the name similarity is greater than or equal to 0.7 and the straight-line distance is less than or equal to 100 meters; or the straight-line distance between two POIs is less than 10 meters and the number of same Chinese characters is greater than or equal to 4, it is considered that they are the same POI;

[0092] S3, when it is other cases, considering the complexity of the collected POI data, in addition to the "same name" and "containing" relationship, there are other various cases between the names of two POIs, which can determine that the two POIs are the same, and the most common ones mainly include three types:

[0093] The first type is that the beginning has more than 3 continuous numbers that are the same and the straight-line distance is less than or equal to 100 meters;

[0094] The second type is that the beginning has more than 4 continuous English letters that are the same and the straight-line distance is less than or equal to 150 meters;

[0095] The third type is to consider the homophonic word situation, when the Chinese pinyin of the names of two POIs is completely the same, it is also considered that they are the same POI;

[0096] S4, in the low-frequency word statistics, using the word segmentation data in the preprocessing process, when the frequency is lower than 75, it is considered as a "low-frequency word", when two POIs have one same "low-frequency word" and the straight-line distance is less than 500 meters, it is further investigated, and there are mainly the following four cases:

[0097] Same "low frequency word" number is greater than or equal to 2, the straight line distance is less than or equal to 100 meters, and the first two characters are the same.

[0098] There is a containing relationship, and the straight line distance is less than or equal to 100 meters.

[0099] The key words in the address after "street", "road" and "number" are highly similar, and the straight line distance is less than or equal to 500 meters.

[0100] When one of them is met, two POIs are determined to be the same.

[0101] The POI data merging step mainly includes the following contents:

[0102] The principle of minimum matching is adopted for fusion: if a POI is identified as a duplicate POI by two methods at the same time, the POI is a duplicate POI, and is removed; if a POI is identified as a duplicate POI by at most one method, the POI is not a duplicate POI, and is retained, then the union of the data sets obtained by removing the same name and different name is taken, and the final result after removing the duplicate is obtained, and the POI is cleaned and removed by combining the name similarity, straight line distance and address similarity multiple similarity indexes, to finally form a high-quality POI fusion library with regular content, rich information and wide coverage, to meet the growing demand for POI. At the same time, more detailed de-duplication rules, automatic rule mining and parameter adjustment of machine learning, more distinguishable POI similarity calculation scheme, POI library without duplicate incremental update, and full-amount POI list slow iteration are all directions for future research.

[0103] Embodiment two:

[0104] The present application is directed to the field of POI fusion and de-duplication, and can also be used in the fields of POI classification, clustering and grid imaging:

[0105] POI similarity calculation, edit distance calculation is to calculate the similarity of POI, and the edit distance of Chinese characters is adopted in the present application, and the edit distance of POI pinyin can also be adopted to calculate the similarity. Other similarity calculation methods are various, including but not limited to SimHash computing Hamming distance, Euclidean distance, Minkowski distance, cosine similarity and the like generated by neural network word vector.

[0106] In the present application, two new similarity calculation schemes based on edit distance and low frequency word are provided. The two calculation schemes can also be used in POI classification, POI clustering analysis, grid-based imaging and other scenes as distance (similarity) calculation schemes.

[0107] The foregoing merely illustrates some exemplary embodiments of the application, and it will be appreciated that those skilled in the art will be able to devise various modifications without departing from the spirit and scope of the application. The appended drawings and description are illustrative only, and are not intended to be limiting.

Claims

1. A method for quickly deduplicating multi-source heterogeneous POI data, characterized in that, It includes three main modules: data preprocessing, raster indexing, and 3x3 grid. Data preprocessing includes the following steps: For POI names, word segmentation, stop word removal, and word frequency statistics are performed to obtain word frequency vectors. Then, the word frequency of the entire database is calculated. For POI coordinates, commonly used POI coordinate systems include GCJ02, WGS84, and BD09. These are uniformly converted to the GCJ02 coordinate system. The raster index includes the following: Raster indexing, to reduce computational load, divides the geographic coordinate system into rectangular grids and establishes an inverse index between the grid and the POI coordinates. The raster index is calculated using the following formula: wherein, is the longitude of the POI, is the latitude of the POI, , are the grid line densities in the east-west and north-south directions, respectively, and finally, , is the grid index coordinate of the POI, and is a floor function.​​​ The nine-grid module comprises two sub-modules: deduplication of POI data with the same name within the nine-grid, deduplication of POI data with different names within the nine-grid, and merging of POI data. The nine-grid module includes the following: Nine-square grid, that is, selecting the center grid For any other raster If grid , The absolute distance between them is less than or equal to 2, determined by the central grid. and all rasters that meet the conditions It is called a "nine-square grid". A "3x3 grid" consists of nine adjacent cells, centered on any one cell, arranged in a grid of nine cells: top, bottom, left, right, top-left, bottom-left, top-right, and bottom-right. If a POI has at least one mirror POI, then the distance between the geographic coordinates of the mirror POI and the coordinates of the original POI must be less than the raster line density, that is, it must fall within the "nine-square grid" centered on the original POI. Therefore, a nine-square grid is a basic deduplication unit. The deduplication of POI data with the same name within the nine-square grid includes the following: For POIs with the same name within the 3x3 grid and a straight-line distance of less than 300m, they are considered the same POI and merged. That is, when two POIs have the exact same name and are close enough, they can be considered the same POI. There are three cases where the names are exactly the same: The first type: When both POI names are a combination of Chinese and English, the Chinese and English parts should correspond to the same values. The second type: When two POI names, one with only Chinese names and the other with a combination of Chinese and English, the corresponding Chinese parts are the same; The third type: When one of the two POI names is only an English name and the other is a combination of Chinese and English, the corresponding English parts are the same.

2. The multi-source heterogeneous POI data fast deduplication method according to claim 1, characterized in that, The deduplication of "different name" POI data within the nine-square grid is achieved using two methods: edit distance and low-frequency word frequency statistics after word segmentation. This includes the following: S1, Edit Distance Edit distance is a quantitative measure of the difference between two strings, measuring the minimum number of operations required to change one string into the other. Let the two strings be and Their lengths are and respectively, and the edit distance between them is and the similarity is According to the meaning of edit distance, the text similarity is defined as follows: wherein On this basis, the name is divided into two categories: there is a containing relationship and other cases. S2, when the names have a containing relationship, the names having a containing relationship are further divided into "sequential containing" and "disordered containing", and the name similarity between two POIs is recorded as , the straight-line distance is , and the number of identical Chinese characters is , and the determination rule is as follows: When two POIs have names that are "sequentially contained", they are considered to be the same POI if the name similarity is greater than or equal to 0.8 and the straight-line distance is less than or equal to 250 meters; or the straight-line distance between the two POIs is less than 30 meters and the number of identical Chinese characters is greater than or equal to 2; or the straight-line distance between the two POIs is less than 50 meters and the number of identical Chinese characters is greater than or equal to 3. When two POIs have names that are "out of order", they are considered to be the same POI if the name similarity is greater than or equal to 0.8 and the straight-line distance is less than or equal to 150 meters; or the name similarity is greater than or equal to 0.7 and the straight-line distance is less than or equal to 100 meters; or the straight-line distance between the two POIs is less than 10 meters and the number of identical Chinese characters is greater than or equal to 4. S3, when other cases, considering the complexity of the collected POI data, two POI names in addition to the "same name", "contains" relationship, there are other various cases to determine two POI as the same, the most common are three types: The first type, the beginning of more than 3 consecutive numbers are the same and the straight line distance is less than or equal to 100 meters; The second type, the beginning of more than 4 consecutive English are the same, and the straight line distance is less than or equal to 150 meters; The third type, considering the homophonic words, when the Chinese pinyin of two POI names are completely the same, also considered as the same POI; S4, in the low frequency word statistics, using the word segmentation data in the preprocessing process, when the frequency is less than 75, it is considered as "low frequency word", when two POIs have one same "low frequency word" and the straight line distance is less than 500 meters, further investigation is carried out, there are four cases: The number of same "low frequency words" is greater than or equal to 2, and the straight line distance is less than or equal to 100 meters The first two characters are the same, and the straight line distance is less than or equal to 100 meters; There is a containing relationship, and the straight line distance is less than or equal to 100 meters; The key words after "street", "road" and "number" in the address are highly similar, and the straight line distance is less than or equal to 500 meters; When one of them is satisfied, it is determined that the two POIs are the same.

3. The multi-source heterogeneous POI data fast deduplication method according to claim 1, characterized in that, The POI data merging step includes the following contents: Take the principle of minimum matching to fuse: if a POI is identified as a duplicate POI by two methods at the same time, the POI is a duplicate POI, which is removed; if a POI is identified as a duplicate POI by at most one method, the POI is not a duplicate POI, which is reserved, then the data set obtained by removing the same name and different name is taken as the union set, and the final result after removing the duplicate is obtained.

Images