String similarity determination

Abstract

String similarity determination. A system and method for determining a similarity between a first string and a second string. A sequence of edit operations to be performed on the first string to obtain the second string can be determined. The edit operations are of a first class or a second class. The first class of operations includes a character insertion operation or a character deletion operation. The second class of operations includes a character preservation operation. The first class of edit operations is associated with an operation score that indicates a cost of applying the edit operation. The first class of edit operations is associated with a switch score that indicates whether the first class edit operation is followed by a second class edit operation. The switch scores and / or operation scores associated with the sequence of edit operations are combined to obtain a combined score that indicates a level of similarity between the first and second strings.

Description

Technical Field

[0001] This invention relates to the field of digital computer systems, and more specifically, to a method for determining the similarity between two strings. Background Technology

[0002] Record linking requires linking elements from a source dataset to relevant data items in a target dataset. This can be achieved by matching records between records in the datasets. Record matching involves calculating the similarity between strings. However, there is a continuous need to improve distance measurements. Summary of the Invention

[0003] Various embodiments provide a method, computer system, and computer program product for determining the similarity between two strings, as described in the independent claims. Advantageous embodiments are described in the dependent claims. The embodiments of this disclosure can be freely combined with each other without mutual exclusion.

[0004] On one hand, the present invention relates to a method for determining the similarity between a string s1 having N1 characters and a string s2 having N2 characters, wherein N1 ≥ 0 and N2 ≥ 0. The method includes:

[0005] a. Provide a distance algorithm, which is configured to be used for:

[0006] i. Receive the first string and the second string;

[0007] ii. Determine a sequence of one or more editing operations to be performed on the characters of the first string in order to obtain the second string, the editing operations being of type I or type II, the type I editing operations including character insertion or character deletion operations, and the type II editing operations including character retention operations; wherein, the type I editing operations are associated with an operation score indicating the cost of applying the editing operations; wherein, the type I editing operations are associated with a switching score indicating whether the type I editing operations in the sequence are immediately followed by the type II editing operations;

[0008] iii. Combine the switching scores and / or operation scores associated with the sequence of the editing operations to obtain a combined score indicating the level of similarity between the first string and the second string;

[0009] b. Input the first n2 characters of string s1 as the first string and the first n2 characters of string s2 as the second string into the distance algorithm to obtain the combined score, where 0≤n1≤N1, 0≤n2≤N2;

[0010] c. Use the obtained combined score to determine the distance between string s1 and string s2.

[0011] On the other hand, the present invention relates to a computer program product comprising a computer-readable storage medium containing computer-readable program code configured to implement all steps of the method according to the foregoing embodiments.

[0012] On the other hand, the present invention relates to a computer system for determining the similarity between a string s1 having N1 characters and a string s2 having N2 characters, wherein N1 ≥ 0 and N2 ≥ 0. The computer system is configured to:

[0013] a. Provide a distance algorithm, which is configured to be used for:

[0014] i. Receive the first string and the second string;

[0015] ii. Determine a sequence of one or more editing operations to be performed on the characters of the first string in order to obtain the second string, the editing operations being of type I or type II, the type I editing operations including character insertion or character deletion operations, and the type II editing operations including character retention operations; wherein, the type I editing operations are associated with an operation fraction indicating the cost of applying the editing operations; wherein, the type I editing operations are associated with a switching fraction indicating whether the type I editing operation in the sequence is immediately followed by a type II editing operation;

[0016] iii. Combine the switching scores and / or operation scores associated with the sequence of the editing operations to obtain a combined score indicating the level of similarity between the first string and the second string;

[0017] b. Input the distance algorithm with the first n1 characters of string s1 as the first string and the first n2 characters of string s2 as the second string to obtain the combined score, where 0≤n1≤N1 and 0≤n2≤N2;

[0018] c. Use the obtained combined score to determine the distance between string s1 and string s2. Attached Figure Description

[0019] The embodiments of the present invention will be explained in more detail below by way of example, with reference to the following figures, wherein:

[0020] Figure 1 This is a block diagram of a computer system that is an example of the subject matter of the present invention.

[0021] Figure 2 This is a flowchart illustrating a method for determining the similarity between two strings according to an example of the subject matter of the present invention.

[0022] Figure 3This is a flowchart illustrating a method for determining the similarity between two strings according to an example of the subject matter of the present invention.

[0023] Figure 4 This is a flowchart illustrating a method for determining the similarity between two strings according to an example of the subject matter of the present invention.

[0024] Figure 5A This is a flowchart illustrating a method for determining the similarity between two strings according to an example of the subject matter of the present invention.

[0025] Figure 5B The evolution of the contents of a matrix indicating edit distance is shown as an example of the subject matter of the invention.

[0026] Figure 6A This is a flowchart illustrating a method for determining the similarity between two strings according to an example of the subject matter of the present invention.

[0027] Figure 6B An example illustrating the evolution of the contents of a matrix indicating the edit distance is shown in the present invention.

[0028] Figure 7 This is an example of pseudocode used to determine the similarity between two strings according to the subject matter of the present invention.

[0029] Figure 8 This refers to a computerized system suitable for implementing one or more method steps involved in the present invention. Detailed Implementation

[0030] The description of various embodiments of the present invention is for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, their practical application relative to technology found in the market, or technical improvements thereof, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0031] The similarity between two strings can be measured by the distance between them. However, accurately determining the distance applicable to a large number of different compared strings can be a challenging task. To address this, the subject matter of this invention provides a string similarity function that provides a numerical value indicating a distance specific to the algorithm of this invention. This value can be the combined score as described herein. The term "string" as used herein can be a sequence of zero or more characters, where the characters can be numbers, letters, or any special characters. Accurately determining the similarity between two strings impacts several application areas where similarity analysis may be used. For example, the string distance algorithm of this invention can be used in fraud detection, fingerprint analysis, plagiarism detection, ontology merging, DNA analysis, RNA analysis, image analysis, evidence-based machine learning, database data deduplication, data mining, incremental search, data integration, malware detection, semantic knowledge integration, and natural language processing, where automatic spell correction can determine candidate corrections for misspelled words by selecting words from a dictionary that are close to related words.

[0032] This invention calculates edit distance by counting the number of editing operations required to transform one string into another. Editing operations are scored based on their type. Furthermore, the invention can also score operations based on the order in which they are applied. This provides accurate comparisons and overcomes the following problems with existing edit distance metrics. For example, the Levenshtein similarity between “Textile” and “Textile Company” is the same as the Levenshtein similarity between “Textile” and “TCeoxm tpialney”, simply because the overall insertion of “Company” has the same weight as letters inserted at random positions. The same applies to “Wachter AG” compared to “Wechsler AG” and “Wachter Bau AG”. Levenshtein similarity is significantly detrimental to word permutations. For example, the distance between "Chaussures-Michel" and "Michel-Chassures" is greater than the distance between "Chaussures-Michel" and "Chic-Chaussures," simply because the substitution requires many insertion and deletion operations.

[0033] The present invention provides a distance algorithm. The distance algorithm can take a first string and a second string as input and determine a sequence of one or more editing operations to be performed on the characters of the first string to obtain the second string. The distance algorithm can apply a score allocation rule to score the sequence of one or more editing operations. The result of the scoring is a combined score. The combined score can be the edit distance between the first string and the second string. Editing operations can be of a first class or a second class. First-class editing operations include character insertion operations (referred to as "I") or character deletion operations (referred to as "D"). Second-class editing operations include character preservation operations (referred to as "M"). For example, character preservation operations are named "editing operations" merely for naming purposes, because character preservation operations may not involve editing. The distance algorithm can apply a score allocation rule to each operation in the sequence of operations. The score allocation rule assigns an operation score indicating the cost of applying a first-class editing operation to the first-class editing operation. For example, the operation score can be equal to 1. The score allocation rule can assign an operation score equal to zero to a second-class editing operation, because a second-class editing operation may not involve editing. In another example, the operation score assigned to a given edit operation can be weighted (e.g., multiplied by a weight) by a predefined weight associated with the character to which the edit operation is applied. Furthermore, if a first-type edit operation is immediately followed by a second-type edit operation, the score allocation rule assigns a switching score p (or penalty) to the first-type edit operation, p = SC, where SC is the value of the penalty. For example, the switching score can be equal to one, or preferably any value less than the sum of the costs of inserting and deleting characters. If a first-type edit operation is immediately followed by a second-type edit operation in a sequence, the switching score can be called a first-type switching score. The fact that a first-type edit operation is immediately followed by a second-type edit operation can be called a first-type switching. The first-type switching score provides a penalty for changing the operation type from first-type to second-type. Alternatively, or additionally, if a first-type edit operation is immediately followed by a preceding second-type edit operation in a sequence, the score allocation rule can assign a switching score p to the first-type edit operation. In this case, the switching score can be called a second-type switching score. The fact that a first-type edit operation is immediately followed by a preceding second-type edit operation in a sequence can be called a second-type switching. The second type of switching fraction can be used. Different implementations of the switching fraction can be used for this purpose. In one implementation example, the switching fraction can be defined as p = w sw ×SC, where w can be... sw Set to a value that enables or disables the switching score. The first-type switching score and the second-type switching score can be correlated with weights w, respectively. sw1 and w sw2 In the context of correlation, when switching scores are enabled, the weight of the first type of switching score can be set to 1, i.e., w. sw1=1, while the second type of switching score can be set to 1 (if considered) or to zero (if not considered), for example, w sw2 =1 or 0. The first type of switching fraction can be p = w sw1 ×SC, the second type of switching score can be p = w sw2 ×SC. Therefore, applying a score allocation rule to a sequence of one or more edit operations can produce a switching score and / or operation score for that sequence of edit operations. A distance algorithm can combine these switching scores and / or operation scores associated with the sequence of edit operations to obtain a combined score or edit distance indicating the level of similarity between the first and second strings. For example, this combination can be performed by summing the scores. The advantage of this distance algorithm is that it can provide an accurate edit distance between the compared strings. Using a combination of switching scores and operation scores, accurate edit distances can be achieved according to the score allocation rule of the subject matter of this invention, regardless of how the sequence of edit operations is determined. Different techniques can be used to obtain the sequence of edit operations. For example, different candidate sequences of edit operations can be determined, and then the candidate sequence with the lowest combined score or the lowest edit distance can be selected. For example, the string "shop" can be obtained from the string "soup" by four deletion operations that remove the characters "s", "o", "u", and "p"; and four insertion operations that insert "s", "h", "o", and "p". The above operation produces a sequence of 8 operations, DDDDIIII, which will give a distance of 8 because the two strings can be transformed by 8 edit operations with an operation score of 1. However, the minimum set of operations that can perform this transformation may have a smaller distance. For example, "shop" can be obtained from "soup" by keeping "s", inserting "h", keeping "o", deleting "u", and keeping "p". This is a sequence of 5 operations, MIMDM, with a smaller distance of 6, for example, four toggle scores and two operation scores, where the toggle score is 1 and the operation score is 1. In another example, a sequence of edit operations can be obtained using known techniques such as the Levenshtein edit distance technique.

[0034] In the following text, the two initial strings to be compared can be referred to as string s1 (N1≥0) with N1 characters and string s2 (N2≥0) with N2 characters. A distance algorithm can be configured to calculate the combined score (distance) of the two input strings (referred to as the first string and the second string), where the first string has n1 characters and the second string has n2 characters. Depending on how the distance algorithm is executed, n1 and n2 can be equal to or not equal to N1 and N2, respectively, where 0≤n1≤N1 and 0≤n2≤N2. For a combined score determined by n1=0 and n2>0, the cost of converting an empty character to n2 characters can be indicated, corresponding to n2 insertion operations. For a combined score determined by n1>0 and n2=0, the cost of converting n1 characters to an empty character can be indicated, corresponding to n1 deletion operations.

[0035] In a first implementation example, the distance algorithm can be configured to directly calculate the similarity between the first and second strings without relying on previously calculated scores. For example, the distance algorithm can calculate a combined score for the two strings independently of any other previously calculated scores. In this case, to determine the similarity between strings s1 and s2, the distance algorithm can be called once with an input pair (first string, second string). For example, by inputting two strings s1 and s2 into the distance algorithm (i.e., the first string is s1 and the second string is s2), the distance algorithm can compare the two strings s1 and s2 (once) to obtain a combined score.

[0036] According to one embodiment, when n1 = N1 and n2 = N2, the obtained combined score indicates the distance between string s1 and string s2. The obtained combined score can be the edit distance between string s1 and string s2. For example, if string s1 contains the character sequence "sp" and the second string s2 contains the character sequence "shop", then the sequence of edit operations to be performed on the character "sp" to obtain the character sequence "shop" is: a hold operation (because "s" is held), two consecutive insertion operations for inserting the character "ho", and a hold operation for holding "p". In this case, the operation order can be MIIM. Note that other techniques can be used to determine the sequence of other operations (I, M, and D). The distance algorithm can apply the score allocation rule to the sequence MIIM. The first hold operation can receive an operation score OS1 = 0 because the first hold operation belongs to the second type of operation. The second operation "I" can receive an operation score OS2 = 1 because the second operation is of the first type. The second operation "I" can further receive a second type switching score w. sw2×SC, because there is a switch from the first operation of type 2 "M" to the second operation of type 1 "I". The third operation "I" will receive an operation score OS3 = 1, because the third operation is of type 1. Furthermore, the third operation can also receive a type 1 switching score w. sw1 ×SC, because the operation immediately following "I" is "M", which is a second-class operation. The last operation receives an operation score OS4 = 0 because it is a second-class operation. A combined score can be obtained, for example, by adding the scores or using other combination techniques; for example, the combined score can equal the sum: w sw2 ×SC+OS1+OS2+OS3+w sw1 ×SC+OS4, where w sw1 =1 and w sw2 =1.

[0037] According to one embodiment, the method further includes providing a character weight to each character of strings s1 and s2, wherein the association of the operation score with the first type of edit operation includes weighting the operation score with the character weights of the characters involved in the first type of edit operation, wherein the association of the first type of switch score with the first type of edit operation includes using a weight w c The switching score p is weighted, and the weight w is used to calculate the weight. c This is a function of the combination score of the subsequence of operations where the last operation is a first-type edit operation and the number of first-type edit operations in the subsequence. For example, this function could be the ratio of the combination score to the number of first-type edit operations, but it is not limited to this, as other functions can also be used. In this embodiment, the second-type switching score may not be applied. Following the example above where s1 = "sp" and s2 = "shop", the combination score can be equal to w. sw2 ×SC+w1×OS1+w2×OS2+w3×OS3+w c ×w sw1 ×SC+w4×OS4, where weights w1 to w4 are the weights associated with the four characters of string s2 respectively, w c It is assigned to the switching score w that has already been assigned. sw1 The combined score of the subsequence of operations preceding the ×SC edit operation, divided by the number of operations in that subsequence, w sw1 =1 and w sw2 =0. w c This can be called average character weight. In practice, since the penalty may depend on the weight of the character to be inserted (or deleted), the penalty may not be pre-allocated if there is a deviation from the diagonal level (the same string) to the off-diagonal level, i.e., w. sw2 = 0. Conversely, when returning from an off-diagonal series to a diagonal series, a penalty can be assigned, i.e., w.sw1 =1. Diagonal and off-diagonal series refer to implementations using matrix iteration. The deviation from the diagonal to the off-diagonal series refers to a type II switch, and the deviation from the off-diagonal to the diagonal series refers to a type I switch.

[0038] According to one embodiment, the distance algorithm is further configured to associate a switching score with each of such a first type of edit operation in the sequence of edit operations—the first type of edit operation immediately following a second type of edit operation in the sequence. Continuing with the example of s1 = "sp" and s2 = "shop" above, according to this embodiment, a switching score w can also be assigned to the second operation "I". sw2 ×SC, because it immediately follows the first operation belonging to the second class. In this case, the combinatorial score can be w. sw2 ×SC+OS1+OS2+OS3+w sw1 ×SC+OS4, where w sw1 =1, w sw2 =1. This switching score w sw2 ×SC can be advantageously used for scores calculated without character weights.

[0039] In one embodiment, N1 ≥ 1 and / or N2 ≥ 1. The similarity level between strings s1 and s2 can be modeled by the following function: in It is the average character weight of strings s1 and s2, where d gl (s1,s2) are the combined scores obtained by the method of this invention, and p is the switching score.

[0040] In a second implementation example where previously computed values ​​can be reused, if the distance algorithm is called iteratively to process the characters of strings s1 and s2, the distance algorithm can use previously computed combination scores. In this case, to determine the distance between strings s1 and s2, the distance algorithm can be called multiple times, where the result of the last iteration can be a combination score indicating the distance / similarity between the initial strings s1 and s2. In this example, the distance algorithm can determine the current sequence of operations for the first and second string pairs based on the sequence previously computed in previous iterations. This can potentially save resources required for otherwise unnecessary repetitive determination operations. In this case, the distance algorithm is called first when the first string has n1 = 0 characters of string s1 and the second string has n2 = 0 characters of string s2, corresponding to two empty characters. The distance algorithm can determine that the distance between these two empty characters is zero. Using values ​​for n1 and n2 starting from zero can be advantageous because it allows setting initial values ​​for the combination scores of string pairs (first string, second string) with values ​​n1 = 0 or n2 = 0. Therefore, according to one embodiment, the method further includes providing, determining, or setting initial values ​​for the combination scores of first and second string pairs having n1 and n2 characters respectively, where n1 = 0 and n2 = 0, 1, ..., N2; or n2 = 0 and n1 = 0, 1, ..., N1. This is in Figure 5A -B indicates that initial values ​​can be provided for the first row and first column of the matrix.

[0041] Therefore, according to one embodiment of the second implementation example, the first n1 characters of string s1 and the first n2 characters of string s2 can be repeatedly input into the distance algorithm, wherein in each iteration, new values ​​for n1 and n2 are selected according to nested loops until n1 = N1 and n2 = N2, where n1 represents the outer loop and n2 represents the inner loop. That is, for a given iteration, n1 can be fixed to a value between 0 and N1, n2 can increment from 0 to N2, then n1 can be fixed to the next value between 0 and N1, and so on. This enables... Figures 5A-5B This second implementation example is implemented within the matrix described in [the document]. In each iteration, the distance algorithm can determine / check whether:

[0042] Previously, a first combined score had been determined (e.g., or set / initialized) for a first string with n1-1 (and n1-1≥0) characters and a second string with n2 characters using a first edit operation sequence, and / or

[0043] The second combined score has previously been determined (e.g., or set / initialized) for the first string with n1 characters and the second string with n2-1 (and n2-1≥0) characters using the second edit operation sequence, and / or

[0044] The third combined score has previously been determined (e.g., or set / initialized) for the first string with n1-1 (and n1-1≥0) characters and the second string with n2-1 (and n2-1≥0) characters using the third edit operation sequence, and the last character of the first string and the second string are the same.

[0045] If all the combined scores being examined have previously been calculated and / or set, the distance algorithm can select the lowest score among the calculated / set combined scores. That is, the selected lowest score can be the first, second, or third combined score. The selected lowest score can be a combined score determined for a selected edit operation sequence, where the selected edit operation sequence is the first, second, or third edit operation sequence whose combined score is selected with the lowest score. The distance algorithm can determine additional operations besides the selected edit operation sequence to be performed in order to obtain the second string of the current iteration from the first string of the current iteration. Therefore, the edit operation sequence used to transform the first string of the current iteration into the second string includes the selected edit operation sequence plus the determined additional operations. The distance algorithm can take the additional operations into account when applying the score assignment rules. If the operations preceding the additional operations in the sequence are of different types, this may result in an operation score and a switching score. The selected lowest score can be combined with the operation score and switching score resulting from the additional operations. This combined score is the edit distance between the first and second strings of the current iteration.

[0046] If at least one of the combined scores being examined has not been previously calculated or set, the distance algorithm may calculate that at least one combined score before selecting the lowest score and determining the operation sequence and edit distance as described above. However, this can only occur if n1=0 or n2=0 for the first and second string pairs (e.g., n1=0 and n2=0 represent null characters) and no corresponding initial or set values ​​have been provided beforehand.

[0047] According to one embodiment, the method further includes storing a combined score calculated for each pair of a first string having n1 = N1 characters and a second string having n2 characters ranging from 0 to N2, and a combined score calculated for each pair of a first string having n1 characters ranging from 0 to N1 and a second string having n2 = N2 characters. The method also includes receiving a request to compare two strings s3 and s4, where s3 = s1 + m1 and s4 = s2 + m2, where m1 and m2 are strings of zero or more characters. A second exemplary implementation of the method can be applied to s3 and s4 using the saved scores. This is achieved by repeatedly feeding the distance algorithm the first n1 characters of string s3 and the first n2 characters of string s4, with n1 and n2 changed to new values ​​in each iteration (as described above). The values ​​of n1 iterate in the range 0…N1, while the values ​​of N2 iterate in the range N2+1…N4 (the right quadrant of the matrix). Then, the values ​​of N1 iterate in the range N1+1…N3, while the values ​​of N2 iterate in the range 0…N4 (the lower two quadrants of the matrix).

[0048] According to one embodiment, the determination of the operation sequence and the association of operation scores and switching scores are performed in parallel at the character level.

[0049] According to one embodiment, a distance algorithm can be used to perform record matching between two records. Record matching involves comparing pairs of attribute values ​​of the two records using the distance algorithm to obtain individual similarity levels of the attributes, and combining the individual similarity levels to determine whether the two records are matching records. The distance algorithm can be performed according to any of the exemplary implementations described above.

[0050] A data record, or record, is a collection of related data items, such as a specific user's name, date of birth, and category. A record represents an entity, where an entity refers to a user, object, or concept whose information is stored in the record. The terms "data record" and "record" are used interchangeably. For example, a data record can be stored as a relational entity in a graph database, where each record can be assigned to a node or vertex of the graph, with attributes such as name and date of birth. In another example, a data record can be a record in a relational database.

[0051] Matching records involves comparing the attribute values ​​of the records. For example, if a record includes a set of attributes from a1 to an, a comparison between two records is performed by comparing n pairs of values ​​for each attribute from a1 to an. Thus, a comparison between two or more records can produce n individual similarity levels, indicating the similarity level of the values ​​for each attribute from a1 to an. The similarity level (or matching level) between the compared records can be a combination of individual similarity levels (e.g., the average). The matching level of two records represents the degree of similarity between the attribute values ​​of the two records. Each of the similarity levels—similarity level, individual similarity level, and word-level similarity—can be provided in the form of a normalized value (e.g., between 0 and 1) or in any other format that enables matching records. If the matching level is higher than a predefined similarity threshold, the two records are considered to match. The deduplication system built upon this invention can then merge these records because they represent the same entity. Merging records is an operation that can be implemented in different ways. For example, merging two records may include creating a golden record in place of visually similar records that have been found to be duplicates of each other. This is known as data fusion or physical collapse with record-level or attribute-level survivability. If the matching level is less than or equal to a predefined similarity threshold, it indicates that the two records do not match and can therefore remain as separate data records.

[0052] In one embodiment, N1 ≥ 1 and / or N2 ≥ 1. The similarity level between string s1 and string s2 can be modeled by the following function: Where p is the switching score, d gl (s1,s2) is a combination fraction.

[0053] Figure 1 An exemplary computer system 100 is illustrated. For example, computer system 100 may be configured to perform master data management and / or data warehousing; for instance, computer system 100 may enable a deduplication system. Computer system 100 includes a data integration system 101 and one or more client systems or data sources 105. Client system 105 may include (e.g., reference to...) Figure 8 The computer system described above. Client system 105 can communicate with data integration system 101 via a network connection, including, for example, a wireless local area network (WLAN) connection, a WAN (wide area network) connection, a LAN (local area network) connection, the Internet, or a combination thereof. Data integration system 101 can control access to the central repository 103 (read / write access, etc.).

[0054] Data records stored in central repository 103 may have values ​​for attribute sets 109A-109P, such as a company name attribute. Although this example is described based on several attributes, more or fewer attributes may be used. Dataset 107 used in accordance with the subject matter of the invention may include at least a portion of the records in central repository 103.

[0055] Data records stored in the central repository 103 can be received from client system 105, processed by data integration system 101, and then stored in the central repository 103. The received records may or may not have the same set of attributes 109A-109P. For example, a data record received by data integration system 101 from client system 105 may not have all values ​​of attribute set 109A-109P; for instance, the data record may have values ​​of a subset of attributes from attribute set 109A-109P and may not have values ​​of the remaining attributes. In other words, the records provided by client system 105 may have different levels of completeness. Completeness is the ratio of the number of attributes in a data record containing data values ​​to the total number of attributes in attribute set 109A-109P. Furthermore, the records received from client system 105 may have a different structure than the stored records in the central repository 103. For example, client system 105 may be configured to provide records in XML, JSON, or other formats to enable the association of attributes with corresponding attribute values.

[0056] In another example, data integration system 101 can import data records from central repository 103 from client system 105 using one or more extract-transform-load (ETL) batch processes, or via Hypertext Transfer Protocol (“HTTP”) communication, or via other types of data exchange.

[0057] For example, data integration system 101 can be configured to process received records to, for example, identify duplicate records. For this purpose, a distance algorithm 120 implementing at least a portion of the method of the present invention can be used. For example, data integration system 101 can use distance algorithm 120 to process data records received from client system 105 in order to find matching records in dataset 107.

[0058] Figure 2 This is a flowchart illustrating a method for determining the similarity between two strings, based on examples from this topic. For illustrative purposes, Figure 2 The method described in [the document] can be used in [the following context] Figure 1 The system shown is an implementation, but not limited to, that described. Distance algorithm 120 can be configured to execute... Figure 2 The method.

[0059] In step 201, a first string and a second string can be received. The first string contains a sequence of n1 characters, and the second string contains a sequence of n2 characters.

[0060] In step 203, a sequence of one or more editing operations to be performed on the characters of the first string in order to obtain the second string can be determined. This sequence of editing operations can be determined in different ways. For example, in the case of a second implementation example of the distance algorithm, it can be done as described in [reference]. Figure 4 The sequence of editing operations can be determined as described above. For example, it can be as follows: Figure 4 As described in steps 403 to 409, the previously calculated score is used to determine the sequence of edit operations. This is particularly advantageous when iteratively calling a distance algorithm to calculate the distance. In the case of a first implementation example of the distance algorithm, it can be done as described in the reference... Figure 3 The sequence of edit operations can be determined as described above. In another example, a known technique such as the Levenshtein edit distance technique can be used to determine the sequence of edit operations.

[0061] In step 205, each operation in the sequence of one or more edit operations is assigned an operation score, and may also be assigned an additional switching score depending on the type of edit operation. For example, if the operation is a first-type edit operation, it can be associated with an operation score indicating the cost of applying the edit operation. Furthermore, if the operation is a first-type edit operation followed immediately by a second-type edit operation, it can also be associated with a switching score. In the case of an iterative implementation of the distance algorithm, previously assigned scores can be used in step 205 instead of being reassigned to previously processed edit operations. For example, if the current iteration's edit operation sequence is "DII", the combined score obtained in the previous iteration for the sequence "DI" can be used to calculate the combined score for "DII" in this iteration.

[0062] In step 207, the toggle score and / or operation score associated with the edit operation sequence can be combined. This produces a combined score indicating the level of similarity between the first and second strings. The combined score can be the edit distance between the first and second strings.

[0063] Figure 3 This is a flowchart illustrating a method for determining the similarity between two strings, as exemplified by the subject matter of this invention. For illustrative purposes, Figure 3 The method described in [the document] can be used in [the following context]. Figure 1 The system shown is implemented, but is not limited to, this implementation.

[0064] In step 301, the two strings s1 and s2 can be the inputs to the distance algorithm. The string s1 with N1 characters can be the first string of the distance algorithm, and the string s2 with N2 characters can be the second string of the distance algorithm.

[0065] In step 303, the distance algorithm can determine the sequence of edit operations used to obtain the second string s2 from the first string s1. For example, this can be performed by processing the first string s1 character by character sequentially. Processing is performed on each current character of the first string s1 by determining the current character subsequence of the first string s1, where the current subsequence includes the first x characters of the first string s1 ending with the current character; for example, if the first string s1 is "abcdef" and the current character is "c", then the determined current subsequence is "abc". Furthermore, the operations to be performed on the current character subsequence of the first string s1 to obtain the corresponding (same length) subsequence of the second string s2 can be determined. For example, to obtain the second string "shop" from the first string "soup", the first subsequence of the character "s" in "soup" can be processed first. This processing will indicate that "s" should be kept, as it is the same as the corresponding subsequence "s" in the second string "shop". The next subsequence of the character "so" associated with the character "o" can be processed to determine the operations required to obtain the corresponding subsequence "sh" in the second string "shop". This results in the insertion of "h", leading to the first string "shoup" being edited. The next subsequence of characters "shou" associated with the character "u" can be processed to determine the operation needed to obtain the corresponding subsequence "shop" for "shop". This results in the deletion of "u", leading to the first string "shop" being edited. The next subsequence of characters "shop" associated with the last character "p" can be processed to determine the operation needed to obtain the corresponding subsequence "shop" for the second string "shop". This results in the retention of "p". Therefore, the determined sequence of operations is a sequence of 5 operations, MIMDM.

[0066] In step 305, the distance algorithm can apply the score allocation rule to each operation in the determined sequence of edit operations. This results in each edit operation in the sequence of edit operations having an operation score, and optionally, an additional toggle score.

[0067] In step 307, the distance algorithm can calculate the edit distance between strings s1 and s2. For example, the edit distance can be the sum of all scores assigned to a given sequence of edit operations.

[0068] In step 309, the edit distance between strings s1 and s2 may be received—for example, as the output of a distance algorithm.

[0069] Figure 4 This is a flowchart illustrating a method for determining the similarity between two strings s1 and s2, as an example of the subject matter of this invention. String s1 has N1 characters, and string s2 has N2 characters. For illustrative purposes, Figure 4 The method described in [the document] can be used in [the following context] Figure 1 The system shown is implemented, but is not limited to, this implementation.

[0070] In step 401, the distance algorithm can accept a first string containing the first n1 characters of string s1 and a second string containing the first n2 characters of string s2, where 0 ≤ n1 ≤ N1 and 0 ≤ n2 ≤ N2. In the first execution of step 401, the distance algorithm can accept a first string containing the first n1 = 0 characters of string s1 and a second string containing the first n2 = 0 characters of string s2. That is, the distance algorithm can accept two empty characters.

[0071] In step 403, the distance algorithm can check whether it has determined or initialized the combined score of the first and second string pairs (named surrounding pairs) in previous iterations—the first and second strings have n′1≥0 and n′2≥0 characters, respectively, where the surrounding pairs (n′1, n′2) can include (n′1=n1, n′2=n2-1) and / or (n′1=n1-1, n′2=n2) pairs. If the last character of the first and second strings is the same, the surrounding pairs (n′1, n′2) can also include (n′1=n1-1, n′2=n2-1) pairs. The pair (n′1, n′2) will not satisfy the condition only if n1=0 or n2=0. If there are one or more pairs of surrounding pairs that have not been processed or initialized with values ​​before (missing pairs), the distance algorithm can determine one or more editing operations for each pair in the missing pairs in step 405 (starting from scratch) to obtain the n′2 character of the pair from the n′1 character of the pair. It can also calculate the combination score for missing pairs. Then, step 407 can be performed.

[0072] If the distance algorithm has already processed the surrounding pairs of the n′1 and n′2 characters, which means that the distance algorithm has already calculated the edit distance of the surrounding pairs of the sequences of characters (n1, n2-1) and / or (n1-1, n2) and / or (n1-1, n2-1), or the pairs have been initialized with values, then step 407 can be executed.

[0073] In step 407, the distance algorithm can select a surrounding pair with the minimum edit distance among the sequences (n1, n2-1) and / or (n1-1, n2) and / or (n1-1, n2-1). The distance algorithm may have determined, in the previous iteration, a sequence of edit operations to be performed on the first n'1 characters of string s1 to obtain the first n'2 characters of string s2 for the selected pair (n′1, n′2) (named the selected edit operation sequence). Therefore, in step 409, the distance algorithm can determine or assume that the sequence of edit operations to be performed on the first n1 characters of string s1 to obtain the first n2 characters of string s2 is the selected edit operation sequence plus an additional edit operation. This additional edit operation may depend on the selected pair (n′1, n′2). For example, if the selected pair is (n′1, n′2) = (n1, n2-1), then this additional edit operation is an insertion operation. If the selected pair is (n′1, n′2) = (n1-1, n2), then this additional editing operation is a delete operation. If the selected pair is (n′1, n′2) = (n1-1, n2-1), then this additional editing operation is a keep operation.

[0074] In step 411, the distance algorithm determines the edit distance between the first n1 characters of string s1 and the first n2 characters of string s2 by applying a score allocation rule to the additional edit operations and the final edit operation of the selected edit operation sequence, thereby generating an additional score. Furthermore, this additional score, combined with the edit distance between the first n′1 characters of string s1 and the first n′2 characters of string s2, can be provided as the edit distance between the first n1 characters of string s1 and the first n2 characters of string s2.

[0075] It can be determined (step 413) whether n1 = N1 and n2 = N2. If so, the edit distance calculated in step 411 can be provided in step 415 as the edit distance between strings s2 and s2. Otherwise, a new value pair (n) can be defined in step 414. 1, The process can proceed from step 401 to step 415 until n1 = N1 and n2 = N2 is reached. n1 and n2 can be incremented in each iteration using nested loops, where n1 represents the outer loop and n2 represents the inner loop.

[0076] Figure 5A This is a flowchart illustrating an example method for determining the similarity between the string s1 = "soup" with N1 = 4 characters and the string s2 = "shop" with N2 = 4 characters, using the second implementation example. To this end, Figure 5AThe method can be implemented using a matrix whose first dimension represents the character "soup" and the second dimension represents the character "shop," but it is not limited to this matrix implementation. Matrix implementations can achieve efficient utilization of processing resources. In practice, this method fills a matrix row by row, thus avoiding storing the entire matrix in memory, but only storing one row and updates to the current row. For example, if the first row of the matrix is ​​currently stored in memory, the cells of the second row can be computed continuously until the second row is fully computed. Next, the second row is in memory, the third row is filled, and so on. For example, suppose the operation score and the switching score are equal to 1.

[0077] In step 501, the distance algorithm can create, for example... Figure 5B The matrix M520A shown is of size (N1+1)×(N2+1). The last N2 columns of the matrix represent the N2 characters of string s2. The last N1 rows of the matrix represent the N1 characters of string s1. The additional first column and first row represent the special character ∈ representing an empty string. The first row represents the cost value of obtaining the first n2 characters of string s2 from the empty character; for example, the cost of obtaining "sho" from the empty character is 3, which corresponds to three insertion operations, each with a cost value of 1. The first column represents the cost value of obtaining the empty character ∈ from the first n1 characters of string s1; for example, the cost of obtaining the empty character from "so" is 2, which corresponds to two deletion operations, each with a cost value of 1. In other words, matrix M is initialized with initial cost values ​​that can be used when comparing strings s1 and s2.

[0078] Each element of matrix M has two corresponding strings: a first string and a second string. For example... Figure 5B As shown, unit M 22 Unit M has first and second string pairs ("s", "s") 23 Unit M has first and second string pairs ("s", "sh") 55 It has first and second string pairs ("soup", "shop"), etc. This can be handled in one iteration, for example, M. 22 To M 55 The second implementation example of the distance algorithm is performed by filling each cell in the cell with a cost value. The cost value in each cell represents the edit distance between the first string and the second string associated with that cell.

[0079] The distance algorithm can be applied to each current unit M. ij (i is the row index, j is the column index) Execute steps 503 to 505—the current cell M ij The corresponding upper unit M i-1,j Left unit M i,j-1 and diagonal unit Mi-1,j-1 (Referred to as surrounding cells in this document) have pre-computed / initialized values ​​(e.g., surrounding cells may include diagonal cells if the characters assigned to row i and column j are the same). For example, in matrix 520A, only cell M... 22 The surrounding cells are filled with values, so the distance algorithm can be derived from cell M. 22 start.

[0080] Therefore, the distance algorithm can start from the unit M with the first string and the second string pair ("s", "s") 22 Let's begin. The distance algorithm can determine the cost of obtaining "s" from "s", which is zero because it involves a hold operation. This cost can be obtained from the cost of retrieving "s" from the surrounding cell M. 22 The three cell values ​​are derived; for example, a distance algorithm can determine the upper cell value M. 12 and left unit value M 21 Equal to one, diagonal unit value M 11 The value is zero. The distance algorithm can determine the distance from M in step 503. 11 To M 22 From M 12 To M 22 And from M 21 To M 22 The cost of moving / advancing, and choose the lowest one. From M 12 Proceed to M 22 The cost equals unit M 12 The cost of adding the additional operation of deleting the character "s" to the cost of M is 1 + 1 = 2. 21 Proceed to M 22 The cost equals unit M 21 The cost of adding the additional operation of inserting the character "s" is 1 + 1 = 2. From M 11 Proceed to M 22 The cost equals unit M 11 The cost of the distance algorithm plus the cost of the additional operation to preserve the character "s" is 0 + 0. Therefore, in step 505, the distance algorithm can move towards unit M. 22 Assign the lowest zero value. In the resulting matrix 520B, cell M 22 The value is marked with an additional "+" sign to indicate whether it is advancing / moving along the diagonal to reach cell M. 22 (That is, the operation is a hold operation). Matrix 520B contains the results after the first execution of the distance algorithm.

[0081] Figure 5B The contents of the matrices for different iterations of the distance algorithm are shown. For example, matrix 520C represents the matrix in processing such as Figure 5B The current unit M shown 34The previous state. (Compared to unit M) 22 Similarly, the distance algorithm can determine the upper cell value M. 24 and left unit value M 33 It equals 3, while the diagonal unit value M 23 The value equals 2. In step 503, the distance algorithm can determine the distance from M. 23 To M 34 From M 24 To M 34 And from M 33 To M 34 Consider the cost of travel and choose the lowest one. From M 24 Proceed to M 34 The cost equals unit M 24 The cost, plus the cost of the additional operation to delete the character "o", is 3 + 1 = 4. From M 33 Proceed to M 34 The cost equals unit M 33 The cost, plus the cost of the additional operation of inserting the character "o", is 3 + 1 = 4. From M 23 Proceed to M 34 The cost equals unit M 23 The cost, plus the zero cost of the additional operation to hold the character "o" and the switching fraction of 1 for switching to the hold operation, is 2+1. Therefore, in step 505, the distance algorithm can assign the minimum value 3 to unit M. 34 Because of M 34 The value is obtained from the corresponding diagonal cell, so in the resulting matrix 520D, the value is marked with an additional symbol "+". Figure 5BThe diagram shows the contents of matrix M520E after the last iteration of the distance algorithm. In one example, the last row and last column of matrix 520E can be saved so that they can be reused if two strings containing "soup" and "shop" are to be compared. For example, to calculate the edit distance between two strings "∈souppap" and "∈shopping" with N3 and N4 characters respectively, a new N3×N4 matrix can be used, where, since the saved rows and columns can be used, only the last 4 columns and last 3 rows of the new matrix representing the two strings can be calculated. For example, this operation can be performed by repeatedly inputting the first n1 characters of the string "∈souppap" and the first n2 characters of the string "∈shopping" into the distance algorithm in each iteration, changing n1 and n2 to new values ​​(as described above). Here, the value of n1 iterates over the range 0...N1, while the value of n2 iterates over the range N2+1...N4 (the right quadrant of the new matrix). Then, the value of n1 iterates over the range N1+1...N3, and the value of n2 iterates over the range 0...N4 (the next two quadrants of the new matrix).

[0082] The distance algorithm can provide the lower right cell M in step 507. 55 The value is used as the edit distance between the string s1 = "soup" and the string s2 = "shop".

[0083] Figure 5A The method of this invention is advantageous for editing words located in the same position. In other words, it is advantageous for words with longer identical letter lengths, i.e., those with longer sequences calculated along the diagonal in matrix M. There are 2 from the top left to the bottom right. |s1|+|s2|-1 up to 3 |s1|+|s2|-1 There are 3 distinct paths (character substitution is always allowed). |s1|+|s2|-1 (There are several possibilities). Therefore, computing all paths and finding the longest diagonal path is NP-complete. Instead, this method can rely on adding a penalty whenever the diagonal series in the matrix changes or changes from the diagonal series when constructing the matrix.

[0084] In another example, using Figure 5A The method used to compare strings s1 = "shop" and s2 = "shopping" yields matrix 520F. Matrix 520F gives the distance between s1 and s2 as 5 (essentially, as...). Figure 5BAs shown, advancing along the diagonal of "shop" adds penalties for deviations and the four insertion operations for "p", "i", "n", and "g". Since the calculated distance between s1 and s2 may be greater than the sum of the lengths of the two strings (|s1| + |s2|), this method avoids this by employing the following approach: If both strings are empty, the similarity is 1. Otherwise, s1 can be assumed to be the shorter string without loss of generality. Then, if all characters in s1 are contained in s2, at most 2|s1| penalties are introduced, and if |s2| ≤ 2|s1|, at most |s2| - 1 penalties are introduced, meaning there are not enough characters to add two penalties for each character in s1.

[0085] Figure 6A The second implementation example is used to determine whether the string s1 = "Durr" with N1 = 4 characters is different from the string with N2 = 5 characters. A flowchart illustrating an example method for calculating similarity between them. Figure 5A Same, Figure 6A The method can represent the characters of "Durr" using its first dimension and its second dimension. The method uses a matrix of characters, but is not limited to this matrix implementation. In this example, it is assumed that the operation score and the first type of switching score are equal to 1 (the second type of switching score will not be used in this example because character weights are involved). The first type of switching score can be weighted using the average character weights. Furthermore, each character of strings s1 and s2 can be associated with its respective weight. This is in Figure 6B The diagram shows that each character is associated with a weight of 10, except for the character "¨", which is associated with a weight of 1. However, in order to assign penalties using matrices in this implementation, each element of the matrix has a record recording the number of insertion and deletion operations assigned and the combination score. For each cell of the matrix, this is represented by the pair [cost / len], where "cost" represents the combination score and "len" is the cumulative number of insertion and / or deletion operations to date. For example, cell M of matrix 620A... 42 The pair [20 / 2] indicates that the number of operations is 2, and the combined score calculated by the distance algorithm for the first string "Dur" and the second string "D" is 20. The number of first-class edit operations is 2 deletion operations because the set of operations to obtain "D" from "Dur" includes one hold operation to keep "D" and two deletion operations to delete "u" and "r". The average character weight can be obtained by dividing the total penalty "cost" by the number of characters "len" and then multiplying by the constant p = w. sw1×SC is used to calculate penalty points. When assigning penalty points, the number of insertion and / or deletion operations "len" and the accumulated penalty points "cost" are reset to zero, because the corresponding penalty points have been integrated into the accumulated cost.

[0086] In step 601, the distance algorithm can create, for example... Figure 6B The matrix M shown is of size (N1+1)×(N2+1), 620A. The last N2 columns of the matrix represent the N2 characters of string s2. The last N1 rows of the matrix represent the N1 characters of string a1. The additional first column and first row represent special characters representing empty strings. The first row represents the cost value of obtaining the first n2 characters of string s2 from the empty character. For example, the cost of obtaining "Du¨" from the empty character is 21, which corresponds to each of the three insertion operations with a cost value of 1 and weights of 10, 10, and 1 respectively, i.e., 10*1+10*1+1*1. The first column represents the cost value of obtaining the empty character from the first n1 characters of string s1. For example, the cost of obtaining the empty character from "Du" is 20, which corresponds to each of the two deletion operations with a cost value of 1 and a weight of 10, i.e., 10*1+10*1. In other words, the matrix is ​​initialized with initial cost values ​​that can be used when comparing strings s1 and s2.

[0087] Each element of matrix M has two corresponding strings: a first string and a second string. For example... Figure 6B As shown, unit M 22 The unit M has a pair ("D", "D") containing the first and second strings. 23 Unit M has a pair of first and second strings ("D", "Du") 56 It has first and second string pairs ("Durr", "Du¨rr"), etc. It can be achieved by processing, for example, unit M in one iteration. 22 To M 56 Each cell is populated with a cost value to perform a distance algorithm. The cost value in each cell indicates the edit distance between the first and second strings associated with that cell.

[0088] The distance algorithm can be applied to each current unit M. ij (i is the row index, j is the column index) Execute steps 603 to 605, for each current cell M ij It has corresponding surrounding units M with pre-calculated / initialized values. i-1,j M i,j-1 M i,j-1 and M i-1,j-1 For example, if the characters assigned to row i and column j are the same, then the surrounding cells can include diagonal cells M. i-1,j-1 .For example, Figure 6BThe diagram shows the state of matrix 620A after multiple iterations from steps 603 to 605. In matrix 620A, the distance algorithm can process unit M in the next iteration from steps 603 to 605. 45 Because it operates line by line.

[0089] For the current unit M ij The distance algorithm can determine the distance from M in step 603. i-1,j-1 To M ij From M i-1,j To M ij And from M i,j-1 To M ij Consider the cost of movement and choose the lowest one. If the characters assigned to row i and column j are the same, then it can be determined / considered from M. i-1,j-1 To M ij The cost of travel. From M i-1,j To M ij The travel cost is equal to that of unit M. i-1,j The cost is the sum of the costs of the additional operations that remove the character assigned to line i. From M i,j-1 To M ij The travel cost is equal to that of unit M. i,j-1 The cost is the cost of the additional operation of inserting the character assigned to column j. From M i-1,j-1 To M ij The travel cost is equal to that of unit M. i-1,j-1 The cost is added to the cost of the additional operation of maintaining the same character assigned to row i and column j. Therefore, in step 605, the distance algorithm can assign the lowest cost value of the determined cost value to cell M. ij If—for example—the lowest travel cost is from M i-1,j-1 To M ij The cost of travel, then the cost of additional operations can be the first type of switching fraction w. sw1 ×SC, the score is determined by the average character weight. Weighted, where [cost, len] is the unit M i-1,j-1 The recording cost and number. That is, unit M. ij The combined score may equal: cost + w_c × w sw1 ×SC.

[0090] For example, with Figure 6B The contents of matrix 620A, the distance algorithm can be determined in step 603 from M 34 To M 45 From M 35 To M 45 And from M 44 To M 45 The travel cost is used to process unit M45 And select the lowest one. From M 35 To M 45 The travel cost is equal to unit M 35 The cost plus the cost of the additional operation to remove the character "r" assigned to line 4. From M 44 To M 45 The travel cost is equal to unit M 44 The cost plus the cost of the additional operation assigned to insert the character "r" in column 5. From M 34 To M 45 The travel cost is equal to unit M 34 The cost is added to the cost of the additional operation that maintains the same character "r" assigned to the 4th and 5th columns, where the cost of the additional operation includes the first type of switching fraction w. sw1 ×SC, because the additional operation is of type 2 and the last operation is of type 1, where the type 1 switching score is used with unit M. 34 The associated average character weights are weighted, i.e., w c =cost / len = 1 / 1 = 1. Therefore, in step 505, the distance algorithm can assign the minimum cost value 2 to cell M. 45 .

[0091] The content of matrix 620B is the result after processing all elements of the matrix. The distance algorithm can provide the lower right element M in step 607. 56 The value, as the string s1 = "Durr" and the string Edit distance between them. Figure 6B Another matrix 620C is shown, which is the result of iterative execution of the distance algorithm between the strings s1 = "Dunst" and s2 = "Du¨rr".

[0092] In one example, multiple similarity measures s1, s2, ..., sn can be combined. This can be done using the following formula: sc = 0.9max(s1, ..., sn) + 0.1min(s1, ..., sn). In this way, different similarity measures can capture different aspects of the similarity between two strings. For example, the Levenshtein function can capture edit distance, while the Jaccard similarity function can handle word substitution. On the other hand, the Jaccard similarity function might return a similarity of 1.0, which may not be necessary, for different strings. Therefore, instead of simply using the maximum value, the function can be combined by combining the maximum and minimum values. Using the method proposed in this disclosure may result in a 7% increase in recall (85% to 92%).

[0093] Figure 7 The pseudocode illustrates elements of an example workflow for a matrix-based string comparison method to compare strings a1 and s2. This method assumes the strings have been preprocessed and that initial values ​​associated with empty string elements (i.e., the values ​​in the first row of the first column of the matrix) have been added to the strings in an appropriate manner. The method also uses two penalty functions, pen() and pend(). The pseudocode uses a matrix implementation to obtain the edit distance between strings s1 and s2. "(\e)row" refers to an empty row, such as the first row of matrix 520A. "prev" refers to the previous character. "cur" refers to the current row of the matrix. "top" refers to the top row. "ch1" and "ch2" refer to characters in strings s1 and s2, respectively, and are assigned to the row and column of the current cell.

[0094] "s2_dist" and "s2_pen" are the distance and penalty of the current cell using the left cell (i.e., the cell in the same row as the current cell but in the left column). lft_dist refers to the distance assigned to the left cell. lft_pen refers to the penalty assigned to the left cell.

[0095] "s1_dist" and "s1_pen" are the distance and penalty of the current cell using the top cell (i.e., the cell in the same column as the current cell but in the top row). top_dist refers to the distance assigned to the top cell. top_pen refers to the penalty assigned to the top cell.

[0096] "d2_dist" and "d2_pen" are the distance and penalty of the current cell using the diagonal cells (i.e., the cells to the left and top of the current cell). d_dist refers to the distance allocated to the diagonal cells. d_pen refers to the penalty allocated to the diagonal cells.

[0097] The penalty refers to the switching score mentioned in this article, and the distance refers to the operation score.

[0098] If the length of the penalty object passed to the function pen() is zero, indicating a transition from diagonal movement to horizontal movement (representing a second type of switch), the function pen() returns a penalty. Depending on whether the weighted version is implemented, the function pen() may return zero or some penalty to be allocated. The function pend() returns the average penalty for the transition from horizontal movement to diagonal movement (representing a first type of switch) captured by the penalty object. Depending on whether the weighted version is implemented, the function pend() returns the cumulative distance in the penalty object divided by the characters in the penalty object. In the weighted version of the code, depending on the penalty that may be generated, it may be beneficial to enforce following the diagonal by changing the condition "ch1 == ch2" to the condition "ch1 == ch2 && d2_dist < s1_dist && d2_dist < s2_dist".

[0099] Figure 8 Represents a general computerized system 800 (such as a data integration system) suitable for implementing at least a portion of the method steps involved in the present disclosure.

[0100] It should be understood that the methods described herein are at least partially non - interactive and are automated by a computerized system (such as a server or an embedded system). However, in an exemplary embodiment, the methods described herein can be implemented in a (partially) interactive system. These methods can be further implemented in software 812, 822 (including firmware 822), hardware (processor) 805, or a combination thereof. In an exemplary embodiment, the methods described herein are implemented as executable programs in software and are executed by a dedicated or general - purpose digital computer (such as a personal computer, a workstation, a minicomputer, or a mainframe computer). Thus, the most general system 800 includes a general - purpose computer 801.

[0101] In an exemplary embodiment, in terms of the hardware architecture, as Figure 8 shown, the computer 801 includes a processor 805, a memory (main memory) 810 coupled to a memory controller 815, and one or more input and / or output (I / O) devices (or peripherals) 10, 845 communicatively coupled through a local input / output controller 835. The input / output controller 835 can be, but is not limited to, one or more buses or other wired or wireless connections known in the art. The input / output controller 835 can have - omitted for brevity - other elements for enabling communication, such as controllers, buffers (cache memories), drivers, repeaters, and receivers. Further, the local interface can include address, control, and / or data connections for enabling proper communication among the above - mentioned components. The I / O devices 10, 845 as described herein generally can include any general - purpose cryptographic card or smart card known in the art.

[0102] Processor 805 is a hardware device for executing software (especially software stored in memory 810). Processor 805 can be any custom or commercially available processor, central processing unit (CPU), auxiliary processor among several processors associated with computer 801, semiconductor-based microprocessor (in the form of a microchip or chipset), macroprocessor, or any device typically used to execute software instructions.

[0103] Memory 810 may include any or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that memory 810 may have a distributed architecture, in which different components are geographically separated but can be accessed by processor 805.

[0104] The software in memory 810 may include one or more separate programs, each program including an ordered list of executable instructions for implementing logical functions (particularly those involved in embodiments of this disclosure). Figure 8 In the example, the software in memory 810 includes instructions 812, such as instructions for managing a database, like a database management system.

[0105] The software in memory 810 should also typically include a suitable operating system (OS) 811. OS 811 essentially controls the execution of other computer programs—such as possible software 812 used to implement the methods described herein.

[0106] The methods described herein can be in the form of source program 812, executable program 812 (object code), script, or any other entity including an instruction set 812 to be executed. If it is a source program, it needs to be translated by a compiler, assembler, interpreter, etc., which may or may not be included in memory 810, in order to operate appropriately in conjunction with OS 811. Furthermore, the method can be written in an object-oriented programming language with data and method classes, or a procedural programming language with routines, subroutines, and / or functions.

[0107] In an exemplary embodiment, a conventional keyboard 850 and mouse 855 may be coupled to an input / output controller 835. Other output devices, such as I / O device 845, may include, for example, but not limited to, printers, scanners, microphones, etc., but not limited to input devices. Finally, I / O devices 10, 845 may further include devices for transmitting both input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, etc. I / O devices 10, 845 may be any general-purpose cryptographic card or smart card known in the art. System 800 may further include a display controller 825 coupled to a display 830. In an exemplary embodiment, system 800 may further include a network interface for coupling to a network 865. Network 865 may be an IP-based network for communication between computer 801 and any external server, client, etc., via a broadband connection. Network 865 transmits and receives data that may be involved in performing some or all of the steps described herein between computer 801 and external system 30. In an exemplary embodiment, network 865 may be a managed IP network managed by a service provider. Network 865 may be implemented wirelessly, for example using wireless protocols and technologies such as WiFi, WiMax, etc. Network 865 may also be a packet-switched network such as a local area network (LAN), wide area network (WAN), metropolitan area network (MAN), Internet network, or other similar type of network environment. Network 865 may be a fixed wireless network, wireless local area network (LAN), wireless wide area network (WAN), personal area network (PAN), virtual private network (VPN), intranet, or other suitable network system, and includes devices for receiving and transmitting signals.

[0108] If the computer 801 is a PC, workstation, intelligent device, etc., the software in the memory 810 may also include a Basic Input / Output System (BIOS) 822. The BIOS is a collection of basic software routines that initialize and test the hardware at startup, start the OS 811, and support data transfer between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 801 starts.

[0109] When computer 801 is running, processor 805 is configured to execute software 812 stored in memory 810 to transfer data to and from memory 810, and to control the operation of computer 801 in general according to the software. The methods described herein and OS 811, in whole or in part (but typically in part), are read by processor 805, may be cached within processor 805, and subsequently executed.

[0110] When the systems and methods described in this article are as follows Figure 8When implemented in software 812, these methods can be stored on any computer-readable medium (such as memory 820) for use by or in conjunction with any computer-related system or method. Memory 820 may include disk storage, such as HDD storage.

[0111] The main contents of this invention provide the following:

[0112] Item 1: A method for determining the distance between a string s1 having N1 characters and a string s2 having N2 characters, where N1 ≥ 0 and N2 ≥ 0, the method comprising:

[0113] a. Provide a distance algorithm, which is configured to be used for:

[0114] i. Receive the first string and the second string;

[0115] ii. Determine a sequence of one or more editing operations to be performed on the characters of the first string in order to obtain the second string, the editing operations being of type I or type II, the type I editing operations including character insertion or character deletion operations, and the type II editing operations including character retention operations; wherein, the type I editing operations are associated with an operation score indicating the cost of applying the editing operations; wherein, the type I editing operations are associated with a switching score indicating whether the type I editing operations in the sequence are immediately followed by the type II editing operations;

[0116] iii. Combine the switching scores and / or operation scores associated with the sequence of the editing operations to obtain a combined score indicating the level of similarity between the first string and the second string;

[0117] b. Input the first n1 characters of string s1 as the first string and the first n2 characters of string s2 as the second string into the distance algorithm to obtain the combined score, where 0≤n1≤N1 and 0≤n2≤N2;

[0118] c. Use the obtained combined score to determine the distance between string s1 and string s2.

[0119] Item 2: According to the method described in Item 1, where, when n1 = N1 and n2 = N2, the obtained combined score represents the distance between string s1 and string s2.

[0120] Item 3: According to the method described in Item 1, where n1 = 0 and n2 = 0;

[0121] The input also includes:

[0122] The first n1 characters of string s1 and the first n2 characters of string s2 are repeatedly input into the distance algorithm, where n1 and n2 are incremented in a nested loop, where n1 represents the outer loop and n2 represents the inner loop.

[0123] The distance algorithm is configured to determine the sequence of edit operations in each iteration by:

[0124] Determine if:

[0125] The first combination score has already been determined using the first edit operation sequence for the first string with n1-1 characters and the second string with n2 characters, and / or

[0126] The second combination score has already been determined using the second edit operation sequence for the first string with n1 characters and the second string with n2-1 characters, and / or

[0127] The third combination score has been determined previously for the first string with n1-1 characters and the second string with n2-1 characters using the third edit operation sequence, and the last character of the first string and the second string are the same;

[0128] If it is determined that it was not previously determined, then determine the combined score of the first, second and third combined scores, and select the lowest score among the determined combined scores;

[0129] Determine the additional operation to be performed in order to obtain the second string from the first string, in addition to one of the first, second, or third edit operation sequences associated with the selected lowest score, wherein the additional operation is an insertion operation if the selected pair is (n1, n2-1), a deletion operation if the selected pair is (n1-1, n2), and a hold operation if the selected pair is (n1-1, n2-1).

[0130] The edit operation sequence includes one of a first, second, or third edit operation sequence associated with the selected lowest score and the determined additional operation;

[0131] In each iteration, the distance algorithm is configured to combine the switching score and / or operation score associated with the edit operation sequence by combining the lowest score with the switching score and / or the operation score associated with the additional operation.

[0132] The distance between strings s1 and s2 is determined using the combined score obtained in the last iteration.

[0133] Item 4: The method according to Item 3 further includes providing initial values ​​for the combination score of the first and second string pairs having n1 and n2 characters respectively, wherein n1 = 0 and n2 = 0, 1, ... N2; or n2 = 0 and n2 = 0 and n1 = 0, 1, ... N1.

[0134] Item 5: The method described in item 3 or 4 further includes:

[0135] Save the combined score calculated for each pair of a first string having n1 = N1 characters and a second string having n2 characters varying from 0 to N2, and the combined score calculated for each pair of a second string having n1 characters varying from 0 to N1 and n2 = N2 characters;

[0136] Receive requests to compare two strings s3 and s4, each with N3 and N4 characters respectively, where s3 = s1 + m1 and s4 = s2 + m2, and m1 and m2 are strings with zero or more characters; and

[0137] The method is repeated by repeatedly inputting the first n1 characters of string s3 and the first n2 characters of string s4 into the distance algorithm, using the saved scores, where the value of n1 iterates over the range 0…N1, and the value of n2 iterates over the range N2+1…N4, then the value of n1 iterates over the range N1+1…N3, and the value of n2 iterates over the range 0…N4.

[0138] Item 6: The method according to any one of items 1 to 5 above further includes providing a character weight to each character in strings s1 and s2, wherein the association of the operation score with the first type of editing operation includes weighting the association score with the character weights of the characters involved in the first type of editing operation.

[0139] Item 7: The method described in any of items 1 to 6 above further includes providing a character weight to each character in string s1 and string s2, wherein the association of the operation score with the first type of editing operation includes weighting the operation score with the character weights of the characters involved in the first type of editing operation, wherein the association of the switching score with the first type of editing operation includes using a weight w c The switching scores are weighted, and the weight w c It is a predefined function that combines the score of the subsequence of operations whose last operation is a first-class edit operation and the number of first-class edit operations in the subsequence.

[0140] Item 8: According to the method described in Item 7, the function is the ratio of the combined score to the number of first-type edit operations in the subsequence.

[0141] Item 9: According to the method described in any of the preceding items 1 to 8, the switching score is referred to as the first type of switching score, and the distance algorithm is further configured to: associate the second type of switching score with the first type of editing operation if any first type of editing operation in the editing operation sequence is immediately followed by a previous second type of editing operation.

[0142] Item 10: The method according to any one of items 1 to 9 above, wherein the determination of the operation sequence and the association of the operation score and the deviation score are performed in parallel at the character level.

[0143] Item 11: The method described in any of items 1 to 5 above, where N1≥1 and N2≥1, wherein the distance is converted into a similarity metric according to the following formula:

[0144]

[0145] Where p is the switching score, d gl (s1,s2) is a combination fraction.

[0146] Item 12: The method described in any of the preceding items 1 to 11, wherein string s1 is shorter than string s2.

[0147] Item 13: The method described in any of items 6 to 10 above, wherein N1 ≥ 1 and N2 ≥ 1, wherein the similarity level is further determined according to the following formula:

[0148]

[0149] in It is the average character weight of strings s1 and s2, where d gl (s1,s2) are the combination fractions, and p is the switching fraction.

[0150] Item 14: The method described in any of the preceding items 1 to 13, wherein the switching score is less than the sum of the operation scores of a character insertion operation and a character deletion operation.

[0151] Item 15: According to any of the methods described in items 1 to 14 above, the distance algorithm is configured to determine the sequence of the one or more edit operations by identifying different candidate sequences of edit operations and selecting the candidate sequence that provides the lowest combined score.

[0152] Item 16: A record matching method comprising comparing attribute value pairs of two records using any of the methods in items 1 to 15 above, thereby generating an individual similarity level of the attributes, and combining the individual similarity levels to determine whether the two records are matching records.

[0153] Item 17: A computer program product comprising a computer-readable storage medium having computer-readable program code contained therein, the computer-readable program code being configured to implement the method of any of the preceding items 1 to 15.

[0154] Item 18: A computer system for determining the similarity between a string s1 having N1 characters and a string s2 having N2 characters, wherein N1 ≥ 0 and N2 ≥ 0, comprising:

[0155] Memory;

[0156] processor;

[0157] A local data storage having computer-executable code stored thereon, wherein the computer-executable code includes processor-executable program instructions to cause the processor to perform a method according to any one of the preceding items 1 to 15.

[0158] This disclosure can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may comprise a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0159] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A more specific, non-exhaustive list of examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or those with protrusions in slots on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0160] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0161] Computer-readable program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing the status information of the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this disclosure.

[0162] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0163] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the computer's processor or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0164] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, thereby causing the instructions to be executed on the computer, other programmable apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0165] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to different embodiments of this disclosure. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, two consecutively shown blocks may actually be completed as a single step, executed simultaneously, substantially simultaneously, in a manner that partially or completely overlaps in time, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

Claims

1. A method for determining the distance between a string s1 having N1 characters and a string s2 having N2 characters, wherein N1≥0 and N2≥0, the method comprising: Provides a distance algorithm that is configured to be used for: Receive the first string and the second string; Determine a sequence of one or more editing operations to be performed on the characters of a first string in order to obtain a second string. The editing operations are of type I or type II. Type I editing operations include character insertion or deletion operations, and type II editing operations include character retention operations. The type I editing operations are associated with an operation score indicating the cost of applying the editing operation. The type I editing operations are also associated with a switching score indicating whether a type II editing operation immediately follows a type I editing operation in the sequence. Combine the switching scores and / or operation scores associated with the sequence of the editing operations to obtain a combined score indicating the level of similarity between the first string and the second string; Input the first n1 characters of string s1 as the first string and the first n2 characters of string s2 as the second string into the distance algorithm to obtain the combined score, where 0≤n1≤N1 and 0≤n2≤N2; Use the obtained combined score to determine the distance between string s1 and string s2. The input also includes: The first n1 characters of string s1 and the first n2 characters of string s2 are repeatedly input into the distance algorithm, where n1 and n2 are incremented in a nested loop, where n1 represents the outer loop and n2 represents the inner loop. The distance algorithm is configured to determine the sequence of edit operations in each iteration by: Determine if: The first combination score has already been determined using the first edit operation sequence for the first string with n1-1 characters and the second string with n2 characters, and / or The second combination score has already been determined using the second edit operation sequence for the first string with n1 characters and the second string with n2-1 characters, and / or The third combination score has been determined previously for the first string with n1-1 characters and the second string with n2-1 characters using the third edit operation sequence, and the last character of the first string and the second string are the same; If it is determined that it was not previously determined, then determine the combined score of the first, second and third combined scores, and select the lowest score among the determined combined scores; Determine the additional operation to be performed in order to obtain the second string from the first string, in addition to one of the first, second, or third edit operation sequences associated with the selected lowest score, wherein the additional operation is an insertion operation if the selected pair is (n1, n2-1), a deletion operation if the selected pair is (n1-1, n2), and a hold operation if the selected pair is (n1-1, n2-1). The edit operation sequence includes one of a first, second, or third edit operation sequence associated with the selected lowest score and the determined additional operation; In each iteration, the distance algorithm is configured to combine the switching score and / or operation score associated with the edit operation sequence by combining the lowest score with the switching score and / or the operation score associated with the additional operation. Specifically, the combined score obtained in the last iteration is used to determine the distance between strings s1 and s2.

2. The method according to claim 1, wherein, When n1 = N1 and n2 = N2, the obtained combined score represents the distance between string s1 and string s2.

3. The method of claim 1, further comprising providing initial values ​​for the combination scores of the first and second string pairs having n1 and n2 characters respectively. in, n1 = 0 and n2 = 0, 1, ..., N2; or n2 = 0 and n1 = 0, 1, ... N1.

4. The method of claim 1, further comprising: Save the combined score calculated for each pair of a first string having n1 = N1 characters and a second string having n2 characters varying from 0 to N2, and the combined score calculated for each pair of a second string having n1 characters varying from 0 to N1 and n2 = N2 characters; The receive comparisons each have N / Two strings s with N4 characters each. / And the request for s4, where s / =s1+m1 and s4=s2+m2, where m1 and m2 are strings of zero or more characters; and By repeatedly inputting string s into the distance algorithm / The method is repeated using the saved fractions for the first n1 characters and the first n2 characters of string s4, where the value of n1 iterates over the range 0…N1, and the value of n2 iterates over the range N2+1…N4. Subsequently, the value of n1 iterates over the range N1+1…N… / The iteration proceeds from 0 to N4, while the value of n2 iterates over the range 0…N4.

5. The method of claim 1, further comprising providing a character weight to each character in strings s1 and s2, wherein, The association between the operation score and the first type of editing operation includes weighting the association score with the character weights of the characters involved in the first type of editing operation.

6. The method of claim 4, further comprising providing a character weight to each character in string s1 and string s2, wherein, The association between the operation score and the first type of editing operation includes weighting the operation score using the character weights of the characters involved in the first type of editing operation. The association between the switching score and the first type of editing operation includes using weight w. c The switching scores are weighted, and the weight w c It is a predefined function that combines the score of the subsequence of operations whose last operation is a first-class edit operation and the number of first-class edit operations in the subsequence.

7. The method according to claim 6, wherein the function is the ratio of the combined score to the number of first-type edit operations in the subsequence.

8. The method of claim 1, wherein the switching score is referred to as a first type of switching score, and the distance algorithm is further configured to: associate a second type of switching score with the first type of editing operation if any first type of editing operation in the sequence of editing operations is immediately followed by a previous second type of editing operation.

9. The method according to claim 1, wherein, The determination of the operation sequence and the association of the operation scores and deviation scores are performed in parallel at the character level.

10. The method of claim 1, wherein N1≥1 and N2≥1, wherein the distance is converted into a similarity metric according to the following formula: Where p is the switching score, d gl (s1,s2) is a combination fraction.

11. The method according to claim 1, wherein, String s1 is shorter than string s2.

12. The method according to claim 5, wherein, N1≥1 and N2≥1, where the similarity level is further determined according to the following formula: in It is the average character weight of strings s1 and s2, where d gl (s1,s2) are the combination fractions, and p is the switching fraction.

13. The method according to claim 1, wherein, The switching score is less than the sum of the operation scores of a character insertion operation and a character deletion operation.

14. The method of claim 1, wherein the distance algorithm is configured to determine the sequence of the one or more edit operations by identifying different candidate sequences of edit operations and selecting the candidate sequence that provides the lowest combined score.

15. A record matching method comprising comparing attribute value pairs of two records using the method of any one of claims 1 to 14 to generate individual similarity levels of the attributes, and combining the individual similarity levels to determine whether the two records are matching records.

16. A computer program product comprising a computer-readable storage medium having computer-readable program code contained therein, the computer-readable program code being configured to implement the method of any one of claims 1 to 14.

17. A computer system for determining the similarity between a string s1 having N1 characters and a string s2 having N2 characters, wherein N1 ≥ 0 and N2 ≥ 0, comprising: Memory; processor; A local data storage having computer-executable code stored thereon, wherein the computer-executable code includes processor-executable program instructions to cause the processor to perform the method according to any one of claims 1 to 14.

Images