Method and system for processing similarity of address text filled by human

Abstract

The embodiment of the present application discloses a kind of artificial address text similarity processing method and system filled in.This embodiment of the present application removes the regular character appearing in address in a reasonable way, so as to remove the error similarity contribution caused by address regular character.Using lower time complexity, the common substring of all more than two lengths in two addresses is calculated.A designed increasing function can map the length sequence of common substring into [0,1] space, and the function reasonably describes the characteristics of human-filled address data.A kind of address clustering algorithm capable of identifying not similar but actually identical address is designed.

Description

[0001] The present application is a divisional application of the application No. CN201810316265.0, published as CN108536657A, with the title of "Address Text Similarity Processing Method and System for Artificial Filling" filed on April 10, 2018. TECHNICAL FIELD

[0002] The present application relates to the field of electronic technology, in particular to an address text similarity processing method and system for artificial filling. BACKGROUND

[0003] Address is an indispensable link for mailing various items. With the development of express industry and e-commerce, various files and goods delivered by mailing have become an indispensable part of people's lives. At the same time, address is not only an important link for mailing various items, but also has a very important significance in user portrait. In user portrait, the importance of address data and address surrounding environment data is very high. In the fields of loan pre-approval, loan post-disconnection repair and anti-fraud, address data plays a very important role in statistical modeling or data mining process.

[0004] When filling address electronically, the user can be guided to perfect the address data through address data matching. However, when filling address manually, the artificial filling has great randomness, which can easily lead to address missing or error. Since the address filling is for the express delivery personnel, it brings great randomness to the user when filling address. This randomness is easy to be identified by human, but computers are powerless. How to enable the computer to identify the non-formatted address in the mass data and perform subsequent corresponding operation has been the focus of data processing and data mining field. For the computer, the address is just a string of characters, and it is very important to determine the similarity between different strings to measure the address similarity. The existing methods mainly include:

[0005] 1. Cosine method:

[0006] This method is to represent the characters of two addresses as vectors, and then determine the similarity between the two vectors according to the cosine value between the two vectors, which is used as the similarity between the two addresses.

[0007] For example, the following three groups of addresses:

[0008] Beijing, Chaoyang District, Xidawang Road, McDonald's

[0009] Beijing, Chaoyang District, Wanghe Bridge West, McDonald's

[0010] Beijing, Chaoyang District, Xidawang Road, McDonald's next to the subway;

[0011] Take the following set of addresses as an example, the effective fields that can be used for comparison after splitting the three addresses are only:

[0012] McDonald's on Xidadao Road

[0013] McDonald's on Wangheqiao Road

[0014] McDonald's near the subway on Xidadao Road

[0015] (1) At this time, the cosine of the angle between "McDonald's on Xidadao Road" and "McDonald's on Wangheqiao Road" is calculated:

[0016] Arrange all the Chinese characters in the two addresses in lexicographical order: [big, when, and, labor, road, McDonald's, bridge, look, west];

[0017] The vectors of the two addresses are calculated as and

[0018] The cosine of the angle between the two vectors is calculated by the following formula:

[0019]

[0020] (2) Then calculate the cosine of the angle between "McDonald's on Xidadao Road" and "McDonald's near the subway on Xidadao Road": the same method as (1) above can be used to calculate the cosine of the angle between the two addresses as cos θ = 0.6.

[0021] However, the disadvantages of this scheme are very obvious:

[0022] For humans, it can be quickly known that "McDonald's on Xidadao Road" and "McDonald's on Wangheqiao Road" cannot be the same address, while "McDonald's on Xidadao Road" and "McDonald's near the subway on Xidadao Road" have a high probability of being the same address. However, the similarity calculated by the cosine of the angle method for the former is still more than 11% higher than that of the latter; Therefore, it can be seen that the cosine of the angle method cannot process similar addresses like humans. This is because the cosine of the angle method only considers the same characters, but does not consider the continuity of the same characters. Xidadao and Wangheqiao West both have the words "West" and "Look", but since the words "West" and "Look" are not together, these two words do not contribute to the similarity. The cosine of the angle method cannot express whether the characters are continuous or not.

[0023] 2. Edit distance method:

[0024] Edit distance represents the minimum number of edits required to convert one string to another. Here, editing refers to the steps required to replace a character in a string with another character, or to insert or delete characters. Take the following set of addresses as an example:

[0025] Tin Sum Road, Tung Po Wai

[0026] Tung Po Wai, Tin Sum Road

[0027] The edit distance of these two addresses reaches the maximum; the similarity is:

[0028] 1-distance / max(length(addr)) = 1-6 / 6 = 0.

[0029] It can be quickly known by human beings that the two addresses are obviously the same address, and the characteristics of the address determine that the order of the address string can be simply exchanged without affecting reading. The edit distance method cannot deal with this situation.

[0030] 3. Dice coefficient method:

[0031] This method understands the string as a set of characters. The Dice coefficient is a method of measuring the similarity of sets, and its formula is:

[0032]

[0033] Take the following set of addresses as an example:

[0034] Xierqizhongke Institute

[0035] Xisanqike Institute

[0036] Xierqizhongkedaxing A

[0037] Among them, the length of the set of "Xierqizhongke Institute" and "Xisanqike Institute" is 6, and the length of the intersection is 4, so the Dice coefficient is:

[0038]

[0039] Among them, the length of the set of "Xierqizhongke Institute" and "Xisanqike Institute" is 6, and the length of the intersection is 4, so the Dice coefficient is:

[0040]

[0041] For human beings, it can be quickly known that "Xierqizhongke Institute" and "Xisanqike Institute" cannot be the same address; while "Xierqizhongke Institute" and "Xisanqike Institute" are the same address, but their Dice coefficients are equal. It can be seen from this that the Dice coefficient method can correct the shortcomings of the exchange order of the edit distance method, but like the cosine of the included angle method, it cannot solve the influence of continuous characters.

[0042] 4. Jaccard similarity method:

[0043] Jaccard similarity method is similar to Dice coefficient method, and is also a measure of set. A formula of Jaccard similarity method is as follows: Wherein, X and Y represent two sets respectively.

[0044] Taking the following address as an example:

[0045] Xierqizhongkei

[0046] Xisanqikei

[0047] Xierqizhongkedaxi A

[0048] Wherein, the length of the union set of the two sets of 'Xierqizhongkei' and 'Xisanqikei' is 8, and the length of the intersection set is 4, so the Jaccard similarity is as follows:

[0049]

[0050] The length of the union set of the two sets of 'Xierqizhongkei' and 'Xierqizhongkedaxi A' is 10, and the length of the intersection set is 5, so the Jaccard similarity is as follows:

[0051]

[0052] It can be seen that the defects of Jaccard similarity method and Dice coefficient method are the same. SUMMARY

[0053] In view of the problems in the prior art, the purpose of embodiments of the present application is to provide a method and system for processing similarity of address texts filled by human, which can more accurately process the similarity between different addresses, so as to improve the accuracy of data processing. Commonly filled addresses by hand are a relatively arbitrary way. Embodiments of the present application excavate and summarize the address filling rules in real data, and can process data according to the characteristics of the manually filled way, so as to improve the accuracy and efficiency of processing address data filled by hand.

[0054] In order to achieve the above purpose, embodiments of the present application provide a method for processing similarity of address texts filled by human, comprising:

[0055] Step A1, acquiring any two addresses in N addresses to be compared, acquiring address regular characters in each address and taking the address regular characters as break characters, so as to stop counting when a break character is encountered when calculating address continuity, thereby dividing each address into multiple substrings to remove the false similarity contribution caused by address regular characters; wherein the address regular characters include at least one of the following: ['district','street', 'road', 'county', 'road', 'town', 'village', 'city', ',', ','].

[0056] Step A2, comparing two addresses to obtain all common substrings between the two addresses, wherein the common substrings are the same strings between the two addresses, wherein each same substring should include at least two characters; wherein all common substrings are obtained by using a self-modified dynamic programming method, so that the time complexity of finding multiple common substrings is the same as that of finding one common substring.

[0057] Further comprising:

[0058] Step B1, converting the similarity of two addresses to the [0, 1] interval in an increasing function manner by using the following formula:

[0059]

[0060] Wherein, |A∩B| con_i is the length of the i-th continuous common substring of address A and address B;

[0061] (1+|A∩B| con_i )*|A∩B| con_i / 2 is the sum of the arithmetic sequence from 1 to the length of the i-th continuous common substring, which weights the continuous string to increase its influence on similarity; this formula supports the feature of semi-disorder of address data, which is not supported by most existing methods.

[0062] Further comprising:

[0063] Step C1, calculating the similarity between any two addresses in the N addresses to be compared, and obtaining a triangular matrix according to the similarities, and the diagonal of the triangular matrix is all 1;

[0064] Step C2, determining the threshold of similarity by using sampling data, so as to determine two addresses with similarity less than the threshold as different addresses, and two addresses with similarity greater than or equal to the threshold as the same address;

[0065] Step C3, extracting each row vector of the triangular matrix, and removing the addresses corresponding to the elements less than the threshold in the row vector;

[0066] Step C4, judging whether there is an intersection between two sets; if there is, merging the two sets, and all addresses in the merged set are the same class of addresses; judging whether there is an intersection in the set, if there is, returning to step C3, if there is not, the step ends.

[0067] Further comprising:

[0068] Splitting the address into a large address and a small address, wherein the large address is an address ≥ district level; wherein the small address is an address < district level;

[0069] Comparing the large address and the small address respectively; if the similarity of the large address is less than the threshold value of the large address, directly returning 0 similarity, otherwise returning the similarity of the small address.

[0070] Meanwhile, the embodiment of the present application also proposes a human-filled address text similarity processing system, comprising: a similarity subsystem, configured to perform the following steps:

[0071] Step A1, obtaining any two addresses in N addresses to be compared, obtaining address regular characters in each address and taking the address regular characters as break characters to stop counting when encountering the break characters when calculating address continuity, thereby dividing each address into multiple substrings to remove the false similarity contribution caused by the address regular characters; wherein the address regular characters include at least one of the following: ['district','street', 'road', 'county', 'road', 'town', 'village', 'city', ',', ','];

[0072] Step A2, comparing two addresses to obtain common substrings between the two addresses, wherein the common substrings are the same strings between the two addresses, and each same substring should include at least two characters; wherein the obtained all common substrings adopt a self-modified dynamic programming method, so that the time complexity of finding multiple common substrings is the same as that of finding one common substring.

[0073] Further comprising: a similarity conversion subsystem;

[0074] The similarity conversion subsystem is configured to convert the similarity of the two addresses to the interval [0, 1] according to the following formula:

[0075]

[0076] Wherein, |A∩B| con_i is the length of the i-th continuous common substring of address A and address B;

[0077] (1+|A∩B| con_i )*|A∩B| con_i / 2 is the summation of the arithmetic sequence from 1 to the length of the i-th continuous common substring, which weights the continuous string to increase its influence on the similarity; this formula supports the feature of semi-disorder of address data, which is not supported by most existing methods.

[0078] Further comprising: a multi-address association subsystem, configured to perform the following steps:

[0079] Step C1, calculate the similarity between any two addresses in the N addresses to be compared, and obtain a triangular matrix according to the similarities, and the diagonal of the triangular matrix is all 1;

[0080] Step C2, determine the threshold of the similarity by using the sampling data, so as to determine two addresses with a similarity less than the threshold as different addresses, and two addresses with a similarity greater than or equal to the threshold as the same address;

[0081] Step C3, extract each row vector of the triangular matrix, and remove the addresses corresponding to the elements less than the threshold in the row vector;

[0082] Step C4, judge whether there is an intersection between two sets; if there is, merge the two sets, and all addresses in the merged set are the same type of address; judge whether there is an intersection in the set, if there is, return to step C3, if not, the step ends.

[0083] Further comprising:

[0084] Split the address into a large address and a small address, wherein the large address is an address greater than or equal to the district level; wherein the small address is an address less than the district level;

[0085] Compare the large address and the small address respectively; if the similarity of the large address is less than the threshold of the large address, return 0 similarity directly, otherwise return the similarity of the small address.

[0086] The beneficial effects of the above technical solutions of the present application are as follows: the above technical solutions propose a method and system for processing the similarity of manually filled address texts, which can more accurately determine the similarity between two addresses, so as to solve the problem of low accuracy of address similarity measurement in the existing data processing method. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 is the initial state transition matrix of two strings in the embodiment of the present application;

[0088] Figure 2 is the state transition matrix after removing the longest common substring in the embodiment of the present application;

[0089] Figure 3 is a schematic diagram of finding a common substring greater than 2;

[0090] Figure 4 is a schematic diagram of the obtained triangular matrix;

[0091] Figure 5 is a schematic diagram of removing the addresses corresponding to the elements less than the threshold in the row vector of Figure 4

[0092] ​Figure 6 Flowchart for splitting large and small addresses. DETAILED DESCRIPTION

[0093] To illustrate the present application, a further detailed description of the present application is given below in connection with the accompanying drawings and specific embodiments.

[0094] The embodiment of the present application proposes a method for processing similarity of manually filled address text, comprising:

[0095] Step A1, obtaining any two addresses in N addresses to be compared, obtaining address regular characters in each address and taking the address regular characters as break characters to stop counting when encountering break characters when calculating address continuity, thereby dividing each address into multiple substrings to remove the false similarity contribution caused by address regular characters; wherein the address regular characters include at least one of the following: ['district','street', 'road', 'county', 'road', 'town', 'village', 'city', ',', ','];

[0096] Step A2, comparing two addresses to obtain all common substrings between the two addresses, wherein the common substring is the same string between the two addresses, and each same substring should include at least two characters.

[0097] Further comprising:

[0098] Step B1, converting the similarity of the two addresses to the [0, 1] interval in an increasing function manner by using the following formula:

[0099]

[0100] Wherein, |A∩B| con_i is the length of the i-th continuous common substring of address A and address B;

[0101] (1+|A∩B| con_i )*|A∩B| con_i / 2 is the sum of the arithmetic sequence of the length of the first i continuous common substring, to weight the continuous string to increase its influence on similarity; this formula supports the feature of semi-ordered address data, which is not supported by most existing methods.

[0102] Further comprising:

[0103] Step C1, calculating the similarity between any two addresses in N addresses to be compared, and obtaining a triangular matrix according to the similarity, and the diagonal line of the triangular matrix is all 1;

[0104] Step C2, determining a threshold of similarity by using the sampling data, so as to determine two addresses with similarity less than the threshold as different addresses, and two addresses with similarity greater than or equal to the threshold as the same address.

[0105] Step C3, extracting each row vector of the triangular matrix, and removing the address corresponding to the element less than the threshold in the row vector.

[0106] Step C4, judging whether there is an intersection between two sets; if yes, merging the two sets, and all addresses in the merged set are the same class of addresses; judging whether there is an intersection in the set, if yes, returning to step C3, and if no, ending the step.

[0107] The method further comprises:

[0108] Splitting the address into a large address and a small address, wherein the large address is an address greater than or equal to a district level; and the small address is an address less than the district level.

[0109] Comparing the large address and the small address respectively; if the similarity of the large address is less than the threshold of the large address, directly returning 0 similarity, otherwise returning the similarity of the small address.

[0110] Meanwhile, the embodiment of the present application further provides an address text similarity processing system filled by human, comprising: a similarity subsystem, configured to perform the following steps:

[0111] Step A1, obtaining any two addresses in N addresses to be compared, obtaining address regular characters in each address, and taking the address regular characters as break characters, so as to stop counting when the break characters are encountered when calculating address continuity, thereby dividing each address into multiple substrings, so as to remove the false similarity contribution caused by the address regular characters; wherein the address regular characters include at least one of the following: ['district','street', 'road', 'county', 'road', 'town', 'village', 'city', ',', ','].

[0112] Step A2, comparing two addresses to obtain common substrings between the two addresses, wherein the common substrings are the same strings between the two addresses, and each same substring should include at least two characters.

[0113] The similarity subsystem further comprises a similarity conversion subsystem.

[0114] The similarity conversion subsystem is configured to convert the similarity of two addresses to the interval [0, 1] according to the following formula:

[0115]

[0116] Wherein, |A∩B| con_iis the length of the i-th consecutive common substring of address A and address B;

[0117] (1+|A∩B| con_i )*|A∩B| con_i is the sum of the arithmetic sequence from 1 to the length of the i-th consecutive common substring, which weights the consecutive string to increase its influence on the similarity; this formula supports the feature of semi-disorder of address data, which is not supported by most existing methods.

[0118] The method further comprises the following steps of:

[0119] Step C1, calculating the similarity between any two addresses in the N addresses to be compared, and obtaining a triangular matrix according to the similarities, wherein the diagonal of the triangular matrix is all 1;

[0120] Step C2, determining a threshold of the similarity by using the sampling data, so as to determine two addresses with a similarity less than the threshold as different addresses, and two addresses with a similarity greater than or equal to the threshold as the same address;

[0121] Step C3, extracting each row vector of the triangular matrix, and removing the addresses corresponding to the elements less than the threshold in the row vector;

[0122] Step C4, judging whether there is an intersection between two sets; if there is, merging the two sets, wherein all addresses in the merged set are the same class of addresses; judging whether there is an intersection in the set, if there is, returning to step C3, and if there is not, ending the step.

[0123] The method further comprises the following steps of:

[0124] Splitting the address into a large address and a small address, wherein the large address is an address greater than or equal to a district level, and the small address is an address less than the district level;

[0125] Comparing the large address and the small address respectively; if the similarity of the large address is less than the threshold of the large address, directly returning 0 similarity, otherwise returning the similarity of the small address.

[0126] Suppose there are three address data A, B and C, in order to let the computer know whether A and B are more similar or A and C are more similar, an intuitive index using numbers to reflect the similarity degree is needed. This index needs to meet the characteristics of address data, that is, to support address order exchange and character continuity to contribute to the similarity. Therefore, the applicant improves the existing longest common substring (LCS) algorithm.

[0127] The LCS algorithm is an algorithm for finding the longest common substring of two strings and is one of the specific applications of the dynamic programming idea. The embodiments of the present invention are modified based on the existing LCS algorithm to make it more suitable for the comparison of address strings.

[0128] Improvement point 1: Only consecutive characters in two or more common substrings will be considered as common substrings. For example, for "Jianguo Road" and "Jianshe Road", if continuity is not considered, the same characters ("Jian", "Road") will contribute 2 / 3 similarity, which is obviously unreasonable.

[0129] Improvement 2: Some common characters in addresses, such as "Province", "City", "Road", "Street", are useless for determining whether addresses are the same but will contribute to similarity. For example, for "Xinhua Road" and "Zhonghua Road", the same characters ("Hua", "Road") are incorrect similarity contributions. However, directly deleting these characters also has problems. For example, deleting the common character "Road" from "Xinhua Road Community" becomes "Xinhua Community"; and deleting the common character "Road" from "Zhonghua Road Community" becomes "Zhonghua Community", then "Hua Community" is an incorrect similarity contribution. Therefore, consider characters ['Community', 'Street', 'Road', 'County', 'Road', 'Town', 'Village', 'City', ',', ','] as break characters, and when encountering these characters during the calculation of the common substring, stop the accumulator for the length of the common substring.

[0130] In the embodiments of the present invention, the Python code for implementing Improvements 1 and 2 is as follows: special = [u'Community', u'Street', u'Road', u'County', u'Road', u'Town', u'Village', u'City', u',', u',']

[0131] # str1 and str2 represent two addresses respectively, and a state transition matrix for dynamic programming is constructed through these two addresses

[0132]

[0133]

[0134] Improvement 3: Since the characteristics of address data are order-insensitive, not only the longest common substring needs to be found, but all common substrings with a length greater than 2 need to be found. To reduce the time complexity, after finding the longest common substring, instead of deleting the longest part and then performing another dynamic programming, directly use the state transition matrix of the first LCS and find all the remaining common substrings with a length greater than 2 through an algorithm. Repeat the above steps until no common substrings with a length greater than 2 can be found.

[0135] Suppose there are two addresses "abcdef" and "abicdekef" respectively. The initial state transition matrix of these two strings is asFigure 1 The longest common substring cde is then crossed out, and the new state transition matrix is as shown in Figure 2 Figure 1 Figure 2 The two matrices as shown in Figure 3 and It is found that the common length of ef is changed from 2 to 1 because e in cde has been crossed out. However, a and b are not affected because they are before the column that is crossed out. By this finding, the direction of the arrow along the last column of the longest common substring that has been found (as shown in

[0136] ) is followed to find the column that is greater than 1, and the value is decreased by 1 along the direction of i+1, j+1 until 0 or the last column of the matrix is reached.

[0137]

[0138]

[0139] The existing longest common substring LCS algorithm finds the longest common substring, while the embodiment of the present application finds all common substrings. Therefore, the LCS algorithm is improved in the embodiment of the present application and can be called AACS, i.e., LCS->ACS (A stands for ALL), and because multiple improvements are made, ACS->AACS (A stands for Advance).

[0140] The python code of the AACS finally integrated is as follows:

[0141]

[0142] After multiple address data are processed by the AACS method, the returned result is the length of all common substrings of two addresses, for example, [4, 2, 2]. Assuming that there are three addresses A, B and C, the similarity of A and B is [5, 3, 2], and the similarity of A and C is [4, 2, 2], which indicates that B is more similar to A than C. This is logical and intuitive.

[0143] However, a set of numbers is not easy to measure and compare by a computer. Moreover, if the two similarities are [7, 2, 2] and [5, 3, 3, 2] respectively, even a human being cannot easily compare them. Therefore, the embodiment of the present application further increases the Adjust-Jaccard similarity algorithm to compress the set of numbers of the similarity obtained by the AACS algorithm into an interval of [0, 1], and the Adjust-Jaccard similarity algorithm is an increasing function.

[0144] ​The advantage of the existing Jaccard similarity algorithm is to support address order transposition, and the disadvantage is that it does not consider continuous action. In the Adjust-Jaccard similarity algorithm of the embodiment of the application, this problem is solved, and the set of the length of the common substring returned by AACS can be compressed into a value in the interval [0, 1] in an increasing function, so as to facilitate comparison.

[0145] The general expression of the existing Jaccard algorithm is: Wherein A and B represent two sets respectively. The general expression of the Adjust-Jaccard similarity algorithm proposed in the embodiment of the application is:

[0146] Adjust-Jaccard:

[0147] Wherein, |A∩B| con_i is the length of the i-th continuous common substring of address A and address B; wherein (1+|A∩B| con_i )*|A∩B| con_i / 2 is the sum of the arithmetic sequence of the length of the first i continuous common substring, so as to weight the continuous string to increase its influence on the similarity.

[0148] The denominator in the above formula is the sum of the numerator and the difference set of the two strings, which is a scaling for compressing the Adjust-Jaccard similarity in the interval [0, 1].

[0149] Use Adjust-Jaccard to verify the example mentioned above that Jaccard is not applicable:

[0150] Xierqizhongke University

[0151] Xisanqike University

[0152] Xierqizhongkedaxing A

[0153] As calculated before, the Jaccard similarity between "Xierqizhongke University" and "Xisanqike University" is 0.5, and the Jaccard similarity between "Xierqizhongke University" and "Xierqizhongkedaxing A" is also 0.5, which is obviously wrong. Looking at the calculation result of Adjust-Jaccard, in order to verify the performance improvement brought by Adjust-Jaccard alone, temporarily do not use the AACS algorithm mentioned in section 4.2.2, otherwise the performance improvement brought by AACS will also be calculated, which will not be convenient for comparison.

[0154] Adjust-Jaccard similarity between "Xierqizhongke University" and "Xisanqike University":

[0155]

[0156] The Adjust-Jaccard similarity of "Xierqi Chinese Academy of Sciences" and "Xierqi Chinese Academy of Sciences A Building":

[0157]

[0158] From the actual effect, the Adjust-Jaccard similarity has been more in line with the actual situation than the Jaccard similarity. Moreover, Adjust-Jaccard can also well support address order exchange, for example:

[0159] The Adjust-Jaccard similarity of "Xierqi Chinese Academy of Sciences" and "Xierqi Chinese Academy of Sciences A Building":

[0160]

[0161] The similarity of the two addresses is 1, reaching the maximum, which is in line with the actual situation.

[0162] Python code of Adjust-Jaccard:

[0163]

[0164]

[0165] In the aforementioned AACS algorithm, the special characters special = [u' district ', u' street ', u' road ', u' county ', u' road ', u' town ', u' township ', u' city ', u', ', u', '] are not involved in the similarity contribution and cannot be deleted in AACS. But when AACS is processed, these characters need to be deleted, otherwise it will increase the denominator, so that Adjust-Jaccard can never reach 1, which is not in line with the actual situation.

[0166] Therefore, when using the Adjust-Jaccard algorithm together with the aforementioned AACS algorithm, the steps include:

[0167] In the previous examples:

[0168] Xierqi Chinese Academy of Sciences

[0169] Xisanqi Academy of Sciences

[0170] Xierqi Chinese Academy of Sciences A Building

[0171] The similarity of the address set of "Xierqi Zhongguancun" and "Xisanqi Academy of Sciences" is also 0.5, which is a relatively high value. The main reason for this problem is that the Adjust-Jaccard algorithm involves the calculation of the similarity of the common individual words with a continuous degree of 1, and the modification of the LCS in AACS can solve this problem.

[0172] The AACS+Adjust-Jaccard similarity of "Xierqi Zhongguancun" and "Xisanqi Academy of Sciences" is 0, because the two addresses have 4 common substrings, but the continuous degree is 1, so the numerator in the Adjust-Jaccard formula is 0.

[0173] The AACS+Adjust-Jaccard similarity of "Xierqi Zhongguancun" and "Xisanqi Academy of Sciences" is still 0.75

[0174] The AACS+Adjust-Jaccard similarity of "Xierqi Zhongguancun" and "Xisanqi Academy of Sciences" is still 1, which shows that AACS+Adjust-Jaccard supports the exchange of order.

[0175] This is very consistent with the actual situation.

[0176] AACS and Adjust-Jaccard integration python code:

[0177] Alongest = get_near(addr1, addr2)

[0178] Similar = _calc_weight_near(Alongest, addr1, addr2)

[0179] At the same time, in reality, there are often such addresses that need to be associated to know that they are the same address, such as:

[0180] Guangdong Province, Guangzhou City, Baiyun District, Tongdewei Street, Yun Da Guangdong Province, Guangzhou City, Baiyun District, Tianshaxia Road, Yun Da Express Guangdong Province, Guangzhou City, Baiyun District, Tongdewei Xisha Road, Yun Da Express

[0181] In the real address data mining process, not only two addresses are given to determine whether they are the same address, usually multiple addresses are given, and these addresses are aggregated into several addresses.

[0182] In order to solve the problem of associated addresses and meet the actual application scene, the embodiment of the present application further puts forward a Similarity Vector Merge algorithm. The steps of the Similarity Vector Merge algorithm specifically include:

[0183] 1. Calculate the similarity between each pair of addresses, the similarity algorithm can use the AACS+Adjust-Jaccard algorithm, and the result can be expressed in an upper triangular matrix with all diagonal elements being 1, as shown in Figure 4 .

[0184] 2. Determine a threshold value of similarity by sampling, a pair of addresses with a similarity less than the threshold value is considered different addresses, otherwise it is considered the same address.

[0185] 3. Take out each row vector of the matrix, and remove the addresses corresponding to the elements in the row vector that are less than the threshold value; as shown in Figure 5 .

[0186] 4. Check if there is an intersection between the two sets. If so, merge the two sets and consider all addresses in the merged set as the same class of address.

[0187] 5. Repeat 3 until there is no intersection between the sets.

[0188] Python code of the Similarity Vector Merge algorithm:

[0189]

[0190]

[0191] The above scheme of the embodiment of the present application is verified by a series of examples:

[0192] After the address similarity measurement index is designed, the most important application scenario is address clustering. Only when the measurement algorithm is designed to meet the actual situation, can the clustering be accurate.

[0193] The remaining two files submitted by the present application demonstrate the address clustering application:

[0194] 1. The addr_dist.py source code file is a summary of the code appearing in each paragraph of this document. The source code includes the implementation of the AACS algorithm, the implementation of the Adjust-Jaccard measurement algorithm, the integration of AACS+Adjust-Jaccard, and the application code of address clustering.

[0195] 2. The testData provides demonstration data, which is a summary of the example addresses appearing in each paragraph of this document. The content is

[0196] Hubei Province, Wuhan City, Wuchang District, University of Science and Technology, Jianhu Campus, Hainian Bookstore

[0197] Hubei Province, Wuhan City, Wuchang District, Gongda Road, Wuhan University of Technology Jianhu Campus Haichen Bookstore

[0198] Hubei Province, Wuhan City, Wuchang District, Gongda Road, Wuhan University of Technology Jianhu Campus

[0199] Shanghai, Shanghai, Baoshan District, Shuichan West Road, Shengzhai Village Primary School

[0200] Shanghai, Baoshan District, Shuichan West Road, Shengzhai Village Primary School

[0201] Guangdong Province, Shenzhen City, Bao'an District, Longhua New Area Guanlan Street, Office

[0202] Guangdong Province, Shenzhen City, Bao'an District, Longhua New Area Guanlan Street, Office

[0203] Guangdong Province, Shenzhen City, Bao'an District, Longhua New Area Guanlan Street, Office

[0204] Guangdong Province, Shenzhen City, Bao'an District, Longhua New Area Guanlan Street, Office

[0205] Beijing, Beijing, Changping District, Xierqi Chinese Academy of Sciences

[0206] Beijing, Beijing, Changping District, Xisanqi Chinese Academy of Sciences

[0207] Beijing, Beijing, Changping District, Xierqi Zhongke Building A

[0208] Beijing, Chaoyang District, Xidawang Road, McDonald's

[0209] Beijing, Chaoyang District, Xidawang Road, McDonald's

[0210] Beijing, Chaoyang District, Xidawang Road, McDonald's

[0211] Guangdong Province, Guangzhou City, Baiyun District, Tongdewei Street, Yun Da

[0212] Guangdong Province, Guangzhou City, Baiyun District, Tianshen West Road, Yun Da Express

[0213] Guangdong Province, Guangzhou City, Baiyun District, Tongdewei West Road, Yun Da Express

[0214] Use a computer that can run python to execute the following command: python addr_dist.py, you can see the return result as follows:

[0215] ====== Group 1 address ======

[0216] Hubei Province, Wuhan City, Wuchang District, University of Technology Jianhu Campus Haichen Bookstore

[0217] Hubei Province, Wuhan City, Wuchang District, Gongda Road, Haiwen Bookstore, Jianhu Campus of Wuhan University of Technology

[0218] Hubei Province, Wuhan City, Wuchang District, Gongda Road, Jianhu Campus of Wuhan University of Technology

[0219] === ...

[0220] Shanghai, Baoshan District, Shuichan West Road, Shengzhai Village Primary School

[0221] Shengzhai Village Primary School, Shuichan West Road, Baoshan District, Shanghai

[0222] === ...

[0223] Guangdong Province, Shenzhen City, Bao'an District, Longhua New District, Guanlan Subdistrict, [Name of Subdistrict Office] @88333333

[0224] Guangdong Province, Shenzhen City, Bao'an District, Longhua New District, Guanlan Subdistrict, [Name of Subdistrict Office] @88000000

[0225] Guangdong Province, Shenzhen City, Bao'an District, Longhua New District, Guanlan Subdistrict, [Name of Subdistrict Office] @88111111

[0226] Guangdong Province, Shenzhen City, Bao'an District, Longhua New District, Guanlan Subdistrict, [Name of Subdistrict Office] @88222222

[0227] === ...

[0228] Beijing, Changping District, Xierqi, Chinese Academy of Sciences

[0229] Beijing, Changping District, Xierqi Zhongke Building A

[0230] === ...

[0231] Beijing, Changping District, Xisanqi Academy of Sciences

[0232] === ...

[0233] McDonald's, Xidawang Road, Chaoyang District, Beijing

[0234] McDonald's next to the subway station on Xidawang Road, Chaoyang District, Beijing.

[0235] === ...

[0236] McDonald's, west of Wanghe Bridge, Chaoyang District, Beijing

[0237] === ...

[0238] Yunda Express, Tianxin Xicha Road, Baiyun District, Guangzhou City, Guangdong Province

[0239] Yunda Express, Tongdewei Xicha Road, Baiyun District, Guangzhou City, Guangdong Province

[0240] Yunda Express, Tongdewei Subdistrict, Baiyun District, Guangzhou City, Guangdong Province

[0241] The above results all conform to the real-world address clustering, indicating that the algorithm design of the address similarity metric is consistent with the actual address filling situation, and the Similarity Vector Merge algorithm is also effective when dealing with multiple address associations.

[0242] appendix

[0243] 1. Size address

[0244] Split the address into a large address and a small address. For example: Shanghai, Shanghai Municipality, Qingpu District, Zhuguang Road, Convention Center. Large address = Shanghai, Shanghai Municipality, Qingpu District;

[0245] Small address: Zhuguang Road, Convention and Exhibition Center

[0246] The reason for this is that comparing large and small addresses together would result in different addresses having a high degree of similarity. For example:

[0247] Shanghai, Shanghai Municipality, Qingpu District, Zhuguang Road, Convention and Exhibition Center

[0248] Shanghai, Shanghai Municipality, Qingpu District, Qingkun Road, Jiaodian Power Plant

[0249] Although the addresses are different, address information at the district level or above contributes approximately 50% of the similarity. Therefore, the addresses should be split into larger and smaller addresses, and then similarity should be judged separately for each. If the similarity of the larger address is less than the threshold for larger addresses, a similarity score of 0 should be returned; otherwise, the similarity score of the smaller address should be returned. The process is as follows: Figure 6 Shown:

[0250] In addition, splitting addresses into large and small addresses offers the added benefit of defining separate similarity thresholds for each. Because large addresses are relatively regular, have high repetition rates, are easily similar prior to each other, and often have fewer characters, the similarity threshold needs to be increased. Conversely, small addresses have the opposite characteristics, so their threshold needs to be decreased.

[0251] Python code for splitting large and small addresses:

[0252] def_get_big_small_addr(addr):

[0253] addr = addr.split(',')

[0254] len_addr = len(addr)

[0255] if len_addr < 3:

[0256] big_addr = addr

[0257] small_addr = None

[0258] elif len_addr > 3:

[0259] big_addr = ','.join(addr[1:3])

[0260] small_addr = ','.join(addr[3:])

[0261] else: # equal to 3

[0262] big_addr = ','.join(addr[1:2])

[0263] small_addr = ','.join(addr[2:])

[0264] return big_addr, small_addr

[0265] addr is filled in the format of "province, city, county, district, road, street, detailed address", and each level of address is separated by ",". But the user may miss any level of data when filling in, so we need to be compatible with various situations.

[0266] References:

[0267] [1] Gusfield, Dan (1999)

[1997] . Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. USA: Cambridge University Press.

[0268] [2] Sidorov, Grigori; Gelbukh, Alexander; Gómez-Adorno, Helena; Pinto, David. "Soft Similarity and Soft Cosine Measure: Similarity of Features in Vector Space Model".

[0269] [3] Levenshtein, Vladimir I. (February 1966). "Binary codes capable of correcting deletions, insertions, and reversals". Soviet Physics Doklady.

[0270] [4] Jaccard, Paul (1912), "The distribution of the flora in the alpine zone", New Phytologist.

[0271] The above describes the preferred embodiments of the present application, it should be noted that for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A method for processing similarity of address text filled in by a person, characterized by, Comprise: Step A1, obtaining any two addresses in N addresses to be compared, obtaining the address regular characters in each address and taking the address regular characters as break characters to stop counting when encountering break characters when calculating address continuity, thereby dividing each address into multiple substrings to remove the error similarity contribution caused by address regular characters and ensuring that no new similarity contribution error is introduced when removing the error similarity contribution caused by regular characters; Wherein the address regular characters include at least one of the following: 'district','street', 'road', 'county', 'road', 'town', 'village', 'city', ',', ', '; Step A2, comparing two addresses to obtain all common substrings between the two addresses, wherein a common substring is a string that is the same between the two addresses, and each identical substring should include at least two characters; wherein the method of obtaining all common substrings uses a self-modified dynamic programming method, so that the time complexity of finding multiple common substrings is the same as that of finding one common substring; Wherein the address text similarity processing method filled in by human beings further comprises: Split the address into large address and small address, wherein the large address is ≥ district level address; wherein the small address is < district level address; Compare the large address and the small address respectively; if the similarity of the large address is less than the threshold value of the large address, return 0 similarity directly, otherwise return the similarity of the small address, wherein the similarity threshold of the large address is greater than the similarity threshold of the small address; Wherein, only continuous characters in 2 or more common substrings are considered as common substrings.

2. The method of claim 1, wherein the address text similarity processing of the artificial filling is characterized by, Further comprise: Step B1, convert the similarity of two addresses to the [0, 1] interval in an increasing function way using the following formula: I = 1...n, n <= min(lenA, lenB) / 2 where |A∩B| is the length of the ith consecutive common substring of address A and address B. con_i where |A∩B| is the length of the ith consecutive common substring of address A and address B. (1 + |A∩B| con_i )*|A∩B| con_i / 2 is the sum of the arithmetic sequence of the length of the first i consecutive common substring, to weight the consecutive string to increase its influence on the similarity; this formula supports the characteristics of semi-ordered address data.

3. The method of claim 2, wherein the address text similarity processing of the artificial filling is characterized by, Further comprise: Step C1, calculate the similarity between any two addresses in N addresses to be compared, and obtain a triangular matrix according to the similarity, and the diagonal of the triangular matrix is all 1; Step C2, determine the threshold value of similarity by using sampling data to determine two addresses with similarity less than the threshold value as different addresses, and two addresses with similarity greater than or equal to the threshold value as the same address; Step C3, extract each row vector of the triangular matrix, and remove the addresses corresponding to the elements less than the threshold value in the row vector; Step C4, judge whether there is an intersection between two sets; if there is, merge the two sets, all addresses in the merged set are the same class of addresses; if there is an intersection in the set, return to step C3, if not, the step ends.

4. A system for processing similarity of address text filled in by a person, characterized by, Comprise: Similarity subsystem for executing the following steps: Step A1, obtaining any two addresses in N addresses to be compared, obtaining address regular characters in each address and taking the address regular characters as break characters to stop counting when a break character is encountered when calculating address continuity, thereby dividing each address into multiple substrings to remove the false similarity contribution caused by address regular characters; wherein the address regular characters include at least one of the following: 'district','street', 'road', 'county', 'road', 'town', 'village', 'city', ',', ','; Step A2, comparing two addresses to obtain common substrings between the two addresses, wherein a common substring is a string that is the same between the two addresses, and each identical substring should include at least two characters; wherein all common substrings are obtained using a self-modified dynamic programming method, so that the time complexity of finding multiple common substrings is the same as that of finding one common substring; Wherein the address text similarity processing system filled in by the human is also used for: Splitting the address into a large address and a small address, wherein the large address is an address ≥ district level; wherein the small address is an address < district level; Comparing the large address and the small address respectively; if the similarity of the large address is less than the threshold value of the large address, then directly return 0 similarity, otherwise return the similarity of the small address, wherein the similarity threshold of the large address is greater than the similarity threshold of the small address; Wherein, only continuous characters in 2 or more common substrings are considered as common substrings.

5. The system for processing similarity of address text filled in by human according to claim 4, wherein, Also includes: A similarity conversion subsystem; The similarity conversion subsystem is used to convert the similarity of two addresses to the [0, 1] interval according to the following formula: i = 1...n, n <= min(lenA, lenB) / 2 where |A∩B| is the length of the i-th consecutive common substring of address A and address B. con_i is the length of the i-th consecutive common substring of address A and address B. (1 + |A∩B| con_i )*|A∩B| con_i / 2 is the sum of the arithmetic sequence of the length of the first i consecutive common substring, to weight the consecutive string to increase its influence on the similarity; this formula supports the characteristics of semi-ordered address data.

6. The system of claim 4, wherein, Also includes: A multi-address association subsystem for performing the following steps: Step C1, calculating the similarity between any two addresses in N addresses to be compared, and obtaining a triangular matrix according to the similarities, and the diagonal of the triangular matrix is all 1; Step C2, determining the threshold value of the similarity using sampling data to determine two addresses with a similarity less than the threshold value as different addresses, and two addresses with a similarity greater than or equal to the threshold value as the same address; Step C3, extracting each row vector of the triangular matrix, and removing the addresses corresponding to the elements less than the threshold value in the row vector; Step C4, determining whether there is an intersection between two sets; if there is, merging the two sets, wherein all addresses in the merged set are the same type of address; determining whether there is an intersection in the set, if there is, returning to step C3, if there is not, the step ends.

Images