A string fuzzy matching method and system

Abstract

The present invention discloses a string fuzzy matching method and system, which relates to the field of character recognition. The method comprises: obtaining multiple characters in different font sizes and constructing a glyph similarity matrix; weighting the cost of the replacement operation by character similarity within the edit distance method to construct a weighted edit distance method; applying a weighted edit distance to the two strings to be matched, normalizing the obtained weighted edit distance, and obtaining the similarity of the two strings to be matched; when the similarity is 1, it indicates a successful match; when the similarity is greater than a set similarity threshold and less than 1, it indicates a suspicious match result; and when the similarity is less than the set similarity threshold, it indicates a failed match. The present invention can solve the problem of character errors in similarity results identified by OCR technology.

Description

Technical Field

[0001] The present invention relates to the field of character recognition, and particularly to a method and system for fuzzy matching of strings. Background Art

[0002] Optical character recognition technology (OCR) can convert different types of documents, such as scanned paper documents, PDF files or images, into editable and searchable data.

[0003] In the prior art, OCR performs fuzzy comparison and search between the recognition result and the standard string by means of image preprocessing, text detection, character segmentation, character recognition and post-processing to find characters similar to the correct characters. In the judgment of similarity, for example, the edit distance method is adopted, which is an algorithm for measuring the similarity between two strings.

[0004] However, although the characters obtained by fuzzy comparison and search are similar to the characters to be recognized, such as "紫" and "柴", they are not the same character, and there is no further distinction between the two similar characters, resulting in incorrect recognition results. Summary of the Invention

[0005] Embodiments of the present invention provide a method and system for fuzzy matching of strings, which can solve the problem of character errors in the similar results recognized by OCR technology in the prior art.

[0006] The embodiment of the present invention provides a string fuzzy matching method, comprising the following steps: obtaining a plurality of characters in different font sizes; selecting any two characters from the plurality of characters, extracting an image feature vector of each character through optical character recognition (OCR) technology, and obtaining a cosine distance of an angle between the image feature vectors of the two characters as a similarity; constructing a glyph similarity matrix based on the similarity between any two characters in the plurality of characters; performing character similarity weighting on the cost of describing the cost of replacing the character in the replacement operation according to a replacement operation for replacing one character of a string with another character in an edit distance method, so as to construct a weighted edit distance method; the character similarity weighting includes: when the similarity of the replaced character is higher than When the similarity threshold is set, the cost of the replacement operation is the difference between the first set value and the similarity; otherwise, the cost of the replacement operation is the second set value, which indicates that a penalty value higher than the difference between the first set value and the similarity is assigned to the cost of the replacement operation; the weighted edit distance method is used for the two strings to be matched. During the replacement operation, the similarity between the two characters that need to be replaced in the two strings to be matched is queried according to the glyph similarity matrix, and the cost of the replacement operation is weighted by the character similarity to obtain the result of the weighted edit distance; the result of the weighted edit distance is normalized to obtain the similarity of the two strings to be matched; the matching result is determined according to the relationship between the similarity of the two strings to be matched and the preset value.

[0007] Furthermore, the weighted edit distance method is used for the two strings to be matched, and the specific steps include:

[0008] Create a ( m +1)×( n +1);

[0009] in, m is the length of the first string to be matched, n is the length of the second string to be matched, the row index of the matrix i The number of characters from left to right corresponding to the first string, the column label of the matrix j Corresponding to the number of characters from left to right in the second string, each matrix element in the matrix D [ i ][ j ] indicates the first character string i Converts the first character of the second string to j The minimum edit distance among the edit distances required for characters;

[0010] Fill the matrix:

[0011]

[0012] in, D [i -1][ j ]+1 is the delete operation, D [ i ][ j -1]+1 is the insertion operation, D [ i -1][ j -1]+cos t For the replacement operation, cos t is the cost of the replacement operation;

[0013] Query the character similarity in the similarity matrix between the character to be replaced in the first string and the character used as a replacement reference in the second string, and weight the cost by character similarity:

[0014]

[0015] in, if Indicates conditions, similarity Indicates character similarity, threshold Indicates the set similarity threshold;

[0016] Filled, the matrix D [ m ][ n ] represents the weighted edit distance between the first character string and the second character string.

[0017] Furthermore, the step of obtaining the cosine distance of the angle between the image feature vectors of two characters as the similarity comprises:

[0018] When two characters are full-width characters, convert the full-width characters to half-width characters;

[0019] The Histogram of Oriented Gradients (HOG) algorithm in the optical character recognition (OCR) technology is used to obtain the HOG descriptor of each character, wherein the HOG descriptor is a one-dimensional array; the similarity comparison between two characters is converted into the similarity comparison between the HOG descriptors of the two characters;

[0020] Get the cosine distance of the angle between the HOG descriptors of two characters as the similarity.

[0021] Furthermore, the query method for the character similarity between the characters to be replaced in the first character string and the characters used as a replacement reference in the second character string in the similarity matrix needs to be optimized to increase the query speed. The optimization specifically includes: if the first character string and the second character string are both company names, the character similarity of all characters in the company name is screened in the similarity matrix to obtain a similarity matrix subset.

[0022] Furthermore, the optimization further includes: when each character in each character string is queried in the similarity matrix, the number of similar characters queried is no more than 15, and the similarity threshold is set to be no less than 0.85.

[0023] Furthermore, the optimization further includes: simultaneously querying, on multiple threads, the character similarity of each character in the first character string and each character in the second character string in the similarity matrix.

[0024] Furthermore, the matching result is determined based on the relationship between the similarity of the two strings to be matched and a preset value, and the specific steps include: when the similarity is 1, it indicates a successful match; when the similarity is greater than the set similarity threshold and less than 1, it indicates a suspicious match result; when the similarity is less than the set similarity threshold, it indicates a failed match.

[0025] Furthermore, the suspicious matching result requires an in-depth comparison of the two character strings to be matched. The specific steps of the in-depth comparison include:

[0026] When the two character strings to be matched are company names, the company names of the two matching character strings are searched separately on a website that can query the unified social credit code, and the search results are obtained and compared;

[0027] If the comparison results are the same, it means the match is successful, and the company names of the two strings to be matched are the same company;

[0028] When the comparison results are different, it indicates that the matching fails, and the corporate names of the two strings to be matched have similar glyphs.

[0029] An embodiment of the present invention provides a string fuzzy matching system, comprising:

[0030] A matrix construction module is used to obtain multiple characters in different font sizes; select any two characters from the multiple characters, extract the image feature vector of each character through optical character recognition technology (OCR), and obtain the cosine distance of the angle between the image feature vectors of the two characters as the similarity; and construct a glyph similarity matrix based on the similarity between any two characters in the multiple characters;

[0031] A weighted edit distance construction and use module is used to perform character similarity weighting on the cost of describing the cost of replacing characters in the replacement operation according to the replacement operation for replacing one character of a string with another character in the edit distance method, so as to construct a weighted edit distance method; the character similarity weighting includes: when the similarity of the replaced characters is higher than the set similarity threshold, the cost of the replacement operation is the difference between the first set value and the similarity; otherwise, the cost of the replacement operation is the second set value, and the second set value indicates that a penalty value higher than the difference between the first set value and the similarity is assigned to the cost of the replacement operation; the weighted edit distance method is used for the two strings to be matched, and during the replacement operation, the similarity between the two characters to be replaced in the two strings to be matched is queried according to the glyph similarity matrix, and the cost of the replacement operation is weighted by character similarity to obtain the result of the weighted edit distance; the result of the weighted edit distance is normalized to obtain the similarity of the two strings to be matched;

[0032] The similarity matching module is used to determine the matching result according to the relationship between the similarity of two character strings to be matched and a preset value.

[0033] The embodiments of the present invention provide a string fuzzy matching method and system. Compared with the prior art, the advantages thereof are as follows:

[0034] Select any two characters from multiple characters, extract the image feature vector of each character through optical character recognition technology OCR, and obtain the angle cosine distance between the image feature vectors of the two characters as the similarity; construct a glyph similarity matrix based on the similarity between any two characters in the multiple characters; according to the replacement operation used to replace one character of a string with another character in the edit distance method, perform character similarity weighting on the cost used to describe the cost of replacing the character in the replacement operation to construct a weighted edit distance method; the character similarity weighting includes: when the similarity of the replaced character is higher than the set similarity threshold, the cost of the replacement operation is first The difference between the set value and the similarity; otherwise, the cost of the replacement operation is the second set value, and the second set value indicates that a penalty value higher than the difference between the first set value and the similarity is assigned to the cost of the replacement operation; the weighted edit distance method is used for the two strings to be matched. During the replacement operation, the similarity between the two characters that need to be replaced in the two strings to be matched is queried according to the glyph similarity matrix, and the cost of the replacement operation is weighted by the character similarity to obtain the result of the weighted edit distance; the result of the weighted edit distance is normalized to obtain the similarity of the two strings to be matched; the matching result is determined according to the relationship between the similarity of the two strings to be matched and the preset value.

[0035] Among them, the cost of the replacement operation increases the consideration of similarity, making the replacement operation affected by similarity. When the similarity of the character to be replaced is higher than the set similarity threshold, the cost of the replacement operation is expressed as the difference between the first set value and the similarity, and the cost is small; otherwise, the cost of the replacement operation is expressed as the second set value, and the cost is high. Then, the weighted edit distance is used for the two strings to be matched, and the replacement operation is affected by the similarity of the characters during the replacement operation. The obtained weighted edit distance is normalized to obtain the similarity of the two strings to be matched. Finally, the improvement of the edit distance method is realized, so that when similar characters are encountered during the comparison process, the similarity of the two characters is considered to distinguish the two similar characters, and the correct recognition result is obtained. Brief Description of the Drawings

[0036] Figure 1 It is a schematic diagram of the glyph similarity matrix of a string fuzzy matching method provided by an embodiment of the present invention. Among them, (a) represents the glyph images of the four characters "Zichaidingxiang" (font = FangSong GB2312, font size = 14), and (b) represents the glyph similarity matrix of the four characters "Zichaidingxiang" (font = FangSong GB2312, font size = 14);

[0037] Figure 2 It is a heat map of the glyph similarity matrix of common characters of a string fuzzy matching method provided by an embodiment of the present invention;

[0038] Figure 3 It is a flowchart of a string fuzzy matching method provided by an embodiment of the present invention. Detailed Embodiment

[0039] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention will be given in conjunction with the accompanying drawings. Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0040] An embodiment of the present invention provides a string fuzzy matching method, including the following steps:

[0041] Step 1: Obtain multiple characters with different font sizes and fonts; select any two characters from the multiple characters, extract the image feature vectors of each character through the optical character recognition technology OCR, and obtain the cosine distance between the image feature vectors of the two characters as the similarity; construct a glyph similarity matrix according to the similarity between any two characters among the multiple characters.

[0042] Step 2: Based on the replacement operation used in the edit distance method to replace one character in a string with another, the cost of the replacement character described in the replacement operation is weighted by character similarity to construct a weighted edit distance method. Character similarity weighting includes: when the similarity of the replaced character is higher than the set similarity threshold, the cost of the replacement operation is the difference between the first set value and the similarity; otherwise, the cost of the replacement operation is the second set value, which indicates that the cost of the replacement operation is penalized by a value higher than the difference between the first set value and the similarity; using the weighted edit distance method for the two strings to be matched, during the replacement operation, the similarity between the two characters to be replaced in the two strings to be matched is queried based on the glyph similarity matrix, and the cost of the replacement operation is weighted by character similarity to obtain the weighted edit distance result; the weighted edit distance result is normalized to obtain the similarity of the two strings to be matched.

[0043] Step 3: Determine the matching result based on the relationship between the similarity of the two strings to be matched and the preset value: when the similarity is 1, it indicates a successful match; when the similarity is greater than the set similarity threshold and less than 1, it indicates a suspicious match result; when the similarity is less than the set similarity threshold, it indicates a failed match.

[0044] The specific operation process is as follows:

[0045] This paper first implements a simple data cleansing function for a list of specialized, innovative, and innovative companies in Excel using custom VBA functions and formulas. Then, using the Histogram of Oriented Gradients (HOG) algorithm, commonly used in optical character recognition (OCR), it extracts the image features of each character. A 3606×3606 glyph similarity matrix is generated and stored between any two characters in a set of commonly used Chinese characters and symbols (2500 common characters, 1000 sub-common characters, and 106 common symbols). Next, a custom weighted Levenshtein algorithm is used to reduce the cost of replacing two similar glyphs (reducing the distance) and impose a penalty factor (increasing the distance) on the replacement of dissimilar glyphs. This enables fuzzy matching of company name strings in OCR recognition scenarios. To improve matching efficiency, the multiprocessing library, which supports parallel computing, is used. For non-perfect matches (non-zero but small edit distance), a code-free visual crawler is designed using the Microsoft Power Automate process automation platform. This crawler searches the Tianyancha website for the unified social credit codes of suspected matching company names to determine whether a match is successful or whether the names are indeed two different companies with particularly similar glyphs.

[0046] The final result is: 760 specialized, sophisticated and new listed companies were completely matched (string similarity = 1, weighted edit distance = 0), 10 specialized, sophisticated and new listed companies were suspiciously matched (string similarity was close to 1, weighted edit distance was close to 0), and by crawling the unified social credit code, it was finally confirmed that 7 were listed companies, 3 were different companies with names very similar to those of listed companies, and there were still 4,152 companies that had no matching results (non-listed specialized, sophisticated and new companies).

[0047] In addition, we conducted robustness tests on the glyph similarity algorithm for common font and size combinations, improved results compared to the traditional edit distance-based character fuzzy matching algorithm, and performance tests.

[0048] 1. Data cleaning and full-width / half-width character conversion.

[0049] To avoid the situation where the mixed use of full-width and half-width characters causes the search and matching to fail, we convert common full-width characters to half-width characters, as shown in Table 1.

[0050] Table 1 Correspondence between full-width and half-width characters

[0051]

[0052] To convert all English letters in the company name string to uppercase, use all half-width characters, and remove titles such as "Stock Co., Ltd.", "Joint Stock Company," and "Limited Company," to reduce the difficulty of subsequent matching, use the following Excel formula (assuming cell A2 contains the original data):

[0053] =UPPER(TRIM(SUBSTITUTE(SUBSTITUTE(SUBSTITUTE(ToHalfWidth(A2),"Stock Co., Ltd.",""),"Stock Co., Ltd.",""),"Limited Liability Company",""))).

[0054] 2. Definition of glyph similarity and calculation of similarity matrix.

[0055] To define glyph similarity, we try to understand it from the perspective of images. If two Chinese characters (with the same font and size) are converted to images, the higher the image similarity, the higher the glyph similarity between the two characters. Therefore, we can use the HOG algorithm (Histogram of Oriented Gradients) to extract features from character images. The HOG algorithm returns a one-dimensional array called a HOG descriptor, which contains feature information for the entire image. Therefore, comparing the similarity between two images becomes comparing the similarity of their HOG descriptors.

[0056] If the resolutions of two images are the same and the HOG parameters used are consistent, then the dimensions (lengths) of the HOG descriptors are also the same. This means that the glyph similarity can ultimately be converted into calculating the cosine of the angle between two high-dimensional vectors (i.e., HOG descriptors), as shown in Equation 1-1.

[0057] (1-1)

[0058] The core code for the above calculation is as follows: The function generate_character_image is used to convert the given text into the corresponding image (the font and font size need to be specified), the extract_features function is used to calculate the HOG feature vector of the given image, and the calculate_similarity_matrix function uses the cosine of the angle to calculate the glyph similarity matrix between all pairs of character sets.

[0059] Taking the four characters "紫", "柴", "丁", and "香" given as examples in the competition questions, the above algorithm can generate images of the four characters, as shown in Figure 1 (a) of. It can be seen that "紫" and "柴" are more similar in glyphs.

[0060] Further calculate the glyph similarity matrix of the above characters, as shown in Figure 1 (b) of. Except that the self-similarity of each character is 1 (diagonal elements), the glyph similarity between "紫" and "柴" is significantly higher than other character combinations, reaching 0.845, which is consistent with the conclusion of subjective observation. This shows that the glyph similarity algorithm designed in this invention is effective and reasonable.

[0061] To compare the glyph similarity, characters, fonts, and font sizes need to be specified. Since all possible situations cannot be exhausted, this invention selects common Chinese characters and characters, and the font size combinations in common scenarios to calculate the glyph similarity matrix for subsequent use. The specific selection reasons are shown in Table 2.

[0062] Table 2 Scenarios and reasons corresponding to the selection of characters, fonts, and font sizes

[0063]

[0064] In Table 2, the following points need to be noted: (1) The complete list of characters used in this invention to generate the similarity matrix is in "\Final Work Submission\1 Fuzzy Matching of Enterprise Name Strings\Step 1_Generate Character Similarity Matrix\Chinese Character Table.xlsx". (2) The file name after the font is the file name corresponding to the font in the submitted source code. (3) To reduce the size of the submitted content, only the similarity matrix of Microsoft YaHei size 14 generated by the program is included in the submitted work file.

[0065] The corresponding file names are similarity_matrix_msyh.ttc_14.xlsx (matrix format) and similarity_list_msyh.ttc_14.csv (list format), and are used for subsequent fuzzy matching. Other result files can be generated by the program if needed.

[0066] The 3606×3606 similarity matrix heat map of common characters under typical configuration (font = Microsoft YaHei, font size = 14) was calculated, as shown in the following figure: Figure 2 As shown in the figure, except for the self-perfected similarity on the main diagonal, the similarity of most characters is low (the blue-green part of the heat map). However, some characters have high similarity (reflected by the red dots outside the main diagonal in the figure). This is the root cause of OCR misrecognition and the basis for subsequent glyph fuzzy matching.

[0067] 3. Weighted Levenshtein distance calculation and string fuzzy matching.

[0068] The traditional Levenshtein distance (also known as the edit distance) is an algorithm used to measure the similarity between two strings. It basically measures similarity by counting the minimum number of edit operations required to transform one string into another. Edit operations include inserting a character, deleting a character, or replacing a character. The steps for calculating the Levenshtein distance include:

[0069] The first step is to initialize the matrix: create a matrix of size (m+1)×(n+1), where m and n are the lengths of the two strings. The row and column indices of the matrix correspond to the number of characters from the left of the first and second strings, respectively. The matrix elements D[i][j] represent the minimum edit distance required to convert the first i characters of the first string to the first j characters of the second string. Initialize the first row and first column of the matrix to increasing values from 0 to m and from 0 to n, respectively. This represents the number of edit operations required to convert a string to an empty string.

[0070] The second step is to fill the matrix: for each element D[i][j] in the matrix, fill it using formula 1-2:

[0071]

[0072] Where cost is the cost of the replacement. After filling, the last element in the matrix (D[m][n]) is the Levenshtein distance between the two strings. For comparability, we often use the normalized edit distance, mapping it to the interval [0, 1] using Equation 1-3.

[0073] (1-3)

[0074] In the traditional Levenshtein distance, if two characters are identical, the cost is 0; otherwise, the cost is 1. This approach fails to account for character similarity. Therefore, this paper incorporates character similarity and defines the cost as Equations 1-4: When the replaced characters are similar (the similarity exceeds a threshold), the cost = 1 - character similarity. Otherwise (the similarity falls below the threshold), a penalty of a larger distance metric (cost = 2) is applied. This approach is more effective at increasing discrimination than the traditional Levenshtein distance, which assigns a cost of 1 to all replacement characters.

[0075]

[0076] Here, similarity represents the similarity between two characters. Similarity is queried from a pre-calculated similarity list. If similarity can be found, the similarity value is returned. If not, it means that the similarity is too low, and -1 is returned. In this case, the cost will increase the penalty weight to 2.

[0077] In addition, since each character comparison requires querying a huge 3606×3606 matrix, the operation speed is slow. When implementing the algorithm, the present invention also makes optimizations in the following aspects to improve the query speed:

[0078] (1) Retain subset: From the complete similarity matrix, pre-screen the similarity matrix subset of all characters involved in the names of "specialized, sophisticated and innovative" enterprises and listed companies, discard other useless information, and reduce the resource overhead of the query.

[0079] (2) Sparse matrix: set the number of similar characters for each character to no more than 15, and the similarity threshold to no less than 0.85, and remove most of the low-value elements in the matrix).

[0080] (3) Parallel computing: Through the multiprocessing library, multiple strings can be compared at once to improve operation efficiency.

[0081] 4. Confirmation of matching results based on the Microsoft Power Automate crawler.

[0082] Fuzzy matching of company names cannot determine whether the small differences between the two strings are due to OCR recognition errors or to two different companies with similar names and glyphs (although humans can make most of these judgments based on experience), as shown in Table 3.

[0083] Table 3 Fuzzy matching results considering glyph similarity (incomplete matching part)

[0084]

[0085] Considering that the unified social credit code can uniquely identify an enterprise, the idea in this part is as follows: send the matched company name to Tianyancha website for query, obtain the unified social credit code of the first company (the company with the most similar name to the query string) in each returned query result, and compare them. If the two are equal, it is the same company and the matching is successful; otherwise, the matching fails.

[0086] Use Microsoft Power Automate to build this code-free crawler. Due to the strict anti-crawler mechanism of Tianyancha, it has been difficult for crawlers based on headless browsers to achieve the expected results. Therefore, use Power Automate to simulate human mouse and keyboard operations to access the website, and use its built-in visual recognition function to determine the position of web page clicks (to avoid triggering the anti-crawler mechanism by traditional crawlers obtaining web page elements) to obtain the value of the unified social credit code.

[0087] Run the crawler, and the final results are shown in Table 4. It can be seen that 3 enterprises finally failed to match, and the listed companies with very similar relevant characters in their names are actually other enterprises. For example, "er" and "le", "yu" and "rong", etc.

[0088] Table 4 Reconfirmation of the results of incomplete matching

[0089]

[0090] Compare the designed standard Levenshtein distance with the optimized Levenshtein distance of the present invention. The time spent on finally matching the list of 4922 specialized, sophisticated, and innovative enterprises is as shown in the following table:

[0091] Table 5 Comparison of the efficiency between the improved algorithm and the traditional algorithm

[0092]

[0093] As can be seen from Table 5, without using the library function to calculate the standard Levenshtein distance, the improved algorithm has achieved a significant improvement in matching accuracy by increasing relatively less time consumption.

[0094] 5. Robustness analysis of the glyph similarity algorithm.

[0095] When calculating the glyph similarity matrix, you need to tell the program which font and font size to use. However, during the fuzzy matching process, information such as which font and font size the string to be matched comes from is generally missing. This requires that the similarity matrix calculated by the algorithm under different fonts and font sizes has a certain degree of stability.

[0096] The correlation coefficient is used to measure the difference between two similarity matrices. The closer the correlation coefficient is to 1, the smaller the impact of font size on the calculated similarity, otherwise it is greater.

[0097] The correlation coefficients of the glyph similarity matrices for different font sizes are all above 0.4. Furthermore, when the font size is small (size 9), the glyph similarity matrices generated by different font sizes vary somewhat. In this case, it is necessary to explicitly specify the font size used in the optical character image to achieve good results. However, for medium and large font sizes, the correlation coefficients are high for all combinations, indicating that the font and size have little influence on the glyph similarity matrix.

[0098] In general, under different font and size combinations, especially when the font size is large, the glyph similarity matrix shows great consistency, and the algorithm is generally robust.

[0099] 6. Comparison between the algorithm of the present invention and the traditional string fuzzy matching algorithm.

[0100] The main difference between the algorithm of the present invention and the traditional string fuzzy matching algorithm is that the similarity of glyphs is taken into account when calculating the Levenshtein distance. In order to analyze the effect of this improvement, we compare the matching accuracy and algorithm efficiency.

[0101] Comparison of matching results:

[0102] Table 6 lists the matching results of the traditional Levenshtein distance when glyph similarity is not considered (only the incomplete matching part is shown), and a unified social credit code verification is performed at the same time. It can be seen that the traditional algorithm has a lot of misrecognitions (FALSE results), while the algorithm of the present invention avoids such misrecognitions to a considerable extent after considering glyph similarity.

[0103] Table 6 String matching results without considering glyph similarity (incomplete matching part)

[0104]

[0105]

[0106] However, it is worth noting that the following three company names that were successfully matched by the traditional algorithm were not matched by the improved algorithm of the present invention. As shown in Table 7, the main reasons involved include: company name changes, errors not caused by OCR recognition, and the presence of uncommon Chinese characters.

[0107] Table 7 Samples and reasons where the traditional algorithm matched successfully but the improved algorithm failed to match

[0108]

[0109] This paper improves the Levenshtein distance measure between character strings based on weighted glyph similarity, proposing a new string fuzzy matching algorithm that is particularly suitable for fuzzy matching of strings that may be subject to OCR misidentification. By matching publicly listed companies in a directory of specialized, innovative, and innovative enterprises identified by OCR, it was found that compared to string fuzzy matching algorithms based on the traditional Levenshtein distance measure, the improved algorithm of the present invention effectively reduces misidentification and balances algorithm efficiency and accuracy. Robustness tests show that the algorithm can adapt to OCR recognition results of different fonts and font sizes.

[0110] An embodiment of the present invention provides a string fuzzy matching system, comprising:

[0111] The matrix construction module is used to obtain multiple characters in different font sizes; select any two characters from the multiple characters, extract the image feature vector of each character through optical character recognition technology (OCR), and obtain the cosine distance of the angle between the image feature vectors of the two characters as the similarity; and construct a glyph similarity matrix based on the similarity between any two characters in the multiple characters.

[0112] A weighted edit distance construction and use module is used to perform character similarity weighting on the cost of describing the cost of replacing characters in the replacement operation according to the replacement operation used to replace one character of a string with another character in the edit distance method, so as to construct a weighted edit distance method; character similarity weighting includes: when the similarity of the replaced character is higher than the set similarity threshold, the cost of the replacement operation is the difference between the first set value and the similarity; otherwise, the cost of the replacement operation is the second set value, and the second set value indicates that a penalty value higher than the difference between the first set value and the similarity is assigned to the cost of the replacement operation; the weighted edit distance method is used for the two strings to be matched, and during the replacement operation, the similarity between the two characters that need to be replaced in the two strings to be matched is queried according to the glyph similarity matrix, and the cost of the replacement operation is weighted by character similarity to obtain the result of the weighted edit distance; the result of the weighted edit distance is normalized to obtain the similarity of the two strings to be matched.

[0113] The similarity matching module is used to determine the matching result according to the relationship between the similarity of the two character strings to be matched and a preset value.

[0114] Innovations of the present invention:

[0115] First, simple data cleaning. This was primarily accomplished in Excel using custom VBA functions and formulas. First, we used Excel formulas to remove titles such as "Company," "Co., Ltd.", "Limited Liability Company," and "Stock Co., Ltd." from the original company names and the public company name table, focusing on matching company names. Second, we wrote a custom VBA function, ToHalfWidth, to convert all full-width characters in a string to their corresponding half-width counterparts in Excel to handle inconsistencies in full-width / half-width brackets and numbers that might appear in the company name list.

[0116] Second, character similarity was calculated. The HOG algorithm, commonly used in OCR recognition, was used to extract the image feature vector of each character, and the cosine distance of the included angle was used to define the character similarity between two characters. Based on this, four commonly used fonts (Microsoft YaHei, Kaiti GB2312, Fangsong GB2312, and Songti) and four commonly used font sizes (Small Five, Small Four, Small Two, and Small First) were selected. The basic data of the 3606×3606 glyph similarity matrix between any two characters from a collection of commonly used Chinese characters and symbols (2500 commonly used characters, 1000 frequently used characters, and 106 commonly used symbols) was calculated and stored under different character size combinations.

[0117] Third, a custom weighted Levenshtein distance is used to calculate string similarity. Based on the basic character similarity data above, by reducing the cost of replacing two similar glyphs (reducing the distance) and applying a penalty factor to the replacement of dissimilar glyphs (increasing the distance), the proposed algorithm ensures that strings incorrectly recognized by OCR have a higher similarity to the correct string, while other strings (with dissimilar characters and very different glyphs) have a lower similarity to the correct string, even if the traditional Levenshtein distance for these two types of strings is equal. Due to the high computational complexity in this step, the multiprocessing library is used to implement parallel computing to improve matching efficiency.

[0118] The unified social credit code was crawled to confirm the fuzzy match results. For non-perfect matches (weighted Levenshtein distance is not zero, but relatively small), due to the strict anti-crawling mechanism of the Tianyancha website, crawling with a headless browser is almost impossible. Therefore, a code-free visual crawler was designed using Microsoft Power Automate to directly simulate human keyboard and mouse operations. It searched the Tianyancha website for unified social credit codes that were suspected to match company names to determine whether they were a match or whether they were indeed two different companies with very similar names.

[0119] Practicality of the present invention:

[0120] This method is used to identify listed companies in a list of "specialized, sophisticated, and innovative" companies that may be subject to OCR misidentification, using a list of standard character string lists for listed company names. Because this problem is essentially a string fuzzy matching problem, it can also be further extended to other string fuzzy matching problems that primarily address glyph similarity scenarios, such as those that may be subject to OCR misidentification.

[0121] A specific embodiment is as follows:

[0122] Step 1: Pre-calculate the glyph similarity matrix (3606×3606) of common Chinese characters and symbols under different font and size combinations based on the HOG algorithm and angle cosine distance, and store it on the hard disk for future use.

[0123] Step 2: Perform simple data cleaning on the matching string list and the standard string list to remove different company names and convert full-width characters to half-width characters.

[0124] Step 3: For each string to be matched, calculate its similarity with each standard string and return the result of the maximum similarity of the string matching and the similarity value. Four inputs are required:

[0125] Input 1: The string to be matched, which in this scenario is the name of a specialized, innovative, and innovative enterprise identified by OCR; Input 2: A list of standard strings, which in this scenario is a list of listed companies; Input 3: A glyph similarity matrix; Input 4: A string similarity threshold, below which a match is considered unsuccessful.

[0126] Step 4: For matches with string similarity = 1, the match is considered successful, and the matched standard string is the matching result; for matches with string similarity ∈ [similarity threshold, 1], the matched standard string is included in the suspicious result for further verification; for matches with string similarity < similarity threshold, the match is considered failed.

[0127] Step 5: For suspicious matching samples, the crawler automatically checks whether the unified social credit code of the first search result of the two matching strings on the Tianyancha website is equal. If they are equal, it is considered a match (the same company); if they are not, it is considered a match failure (two different companies). This also means that the characters of the two company names are very similar.

[0128] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A string fuzzy matching method, characterized in that: The following steps are involved: Get multiple characters in different font sizes; Select any two characters from a plurality of characters, extract the image feature vector of each character through optical character recognition (OCR), and obtain the cosine distance of the angle between the image feature vectors of the two characters as the similarity; construct a glyph similarity matrix based on the similarity between any two characters in the plurality of characters; According to a replacement operation for replacing one character of a string with another character in the edit distance method, a cost describing the cost of replacing the character in the replacement operation is weighted by character similarity to construct a weighted edit distance method; The character similarity weighting includes: when the similarity of the replaced character is higher than a set similarity threshold, the cost of the replacement operation is the difference between the first set value and the similarity; otherwise, the cost of the replacement operation is a second set value, wherein the second set value indicates that a penalty value higher than the difference between the first set value and the similarity is assigned to the cost of the replacement operation; A weighted edit distance method is used for the two strings to be matched. During the replacement operation, the similarity between the two characters to be replaced in the two strings to be matched is queried based on the glyph similarity matrix. The cost of the replacement operation is weighted by the character similarity to obtain the weighted edit distance result. The weighted edit distance result is normalized to obtain the similarity of the two strings to be matched. Determine the matching result based on the relationship between the similarity of the two character strings to be matched and a preset value; The weighted edit distance method is used for the two strings to be matched, and the specific steps include: Create a (m+1)×(n+1) matrix; Where m is the length of the first string to be matched, n is the length of the second string to be matched, the row index i of the matrix corresponds to the number of characters from left to right in the first string, the column index j of the matrix corresponds to the number of characters from left to right in the second string, and each matrix element D[i][j] in the matrix represents the minimum edit distance among the edit distances required to convert the first i characters of the first string to the first j characters of the second string; Fill the matrix: Among them, D[i-1][j]+1 is the deletion operation, D[i][j-1]+1 is the insertion operation, D[i-1][j-1]+cost is the replacement operation, and cost is the cost of the replacement operation; Query the character similarity in the similarity matrix between the character to be replaced in the first string and the character used as a replacement reference in the second string, and weight the cost by character similarity: Among them, if represents the condition, similarity represents the character similarity, and threshold represents the set similarity threshold; After filling, the value of the matrix element D[m][n] in the matrix represents the weighted edit distance between the first character string and the second character string.

2. A string fuzzy matching method as claimed in claim 1, characterized in that: The method of obtaining the angle cosine distance between the image feature vectors of two characters as the similarity comprises the following steps: When two characters are full-width characters, convert the full-width characters to half-width characters; The HOG descriptor of each character is obtained using the Histogram of Oriented Gradients (HOG) algorithm in the optical character recognition (OCR) technology, wherein the HOG descriptor is a one-dimensional array; the similarity comparison between two characters is converted into the similarity comparison between the HOG descriptors of the two characters; Get the cosine distance of the angle between the HOG descriptors of two characters as the similarity.

3. A string fuzzy matching method as claimed in claim 1, characterized in that: The query method for the similarity between the characters to be replaced in the first character string and the characters used as a replacement reference in the second character string in the similarity matrix needs to be optimized to increase the query speed. The optimization specifically includes: If the first character string and the second character string are both company names, the character similarities of all characters in the company name are screened in the similarity matrix to obtain a similarity matrix subset.

4. A string fuzzy matching method as claimed in claim 3, characterized in that: The optimization further includes: setting that when each character in each character string is queried in the similarity matrix, the number of similar characters queried is no more than 15, and the similarity threshold is set to be no less than 0.

85.

5. A string fuzzy matching method as claimed in claim 3, characterized in that: The optimization further includes: simultaneously querying, on multiple threads, the character similarity of each character in the first character string and each character in the second character string in the similarity matrix.

6. A string fuzzy matching method as claimed in claim 1, characterized in that: The step of determining the matching result based on the relationship between the similarity of the two character strings to be matched and the preset value includes: When the similarity is 1, it means the match is successful; When the similarity is greater than the set similarity threshold and less than 1, it indicates a suspicious matching result; When the similarity is less than the set similarity threshold, it means that the matching fails.

7. A string fuzzy matching method as claimed in claim 6, characterized in that: The suspicious matching result requires an in-depth comparison of the two strings to be matched. The specific steps of the in-depth comparison include: When the two character strings to be matched are company names, the company names of the two matching character strings are searched separately on a website that can query the unified social credit code, and the search results are obtained and compared; If the comparison results are the same, it means the match is successful, and the company names of the two strings to be matched are the same company; When the comparison results are different, it indicates that the matching fails, and the corporate names of the two strings to be matched have similar glyphs.

8. A string fuzzy matching system, characterized in that: include: A matrix construction module is used to obtain multiple characters in different font sizes; select any two characters from the multiple characters, extract the image feature vector of each character through optical character recognition technology (OCR), and obtain the cosine distance of the angle between the image feature vectors of the two characters as the similarity; and construct a glyph similarity matrix based on the similarity between any two characters in the multiple characters; A weighted edit distance construction and use module is used to perform character similarity weighting on the cost of replacing a character in a replacement operation for replacing one character in a string with another character in the edit distance method, so as to construct a weighted edit distance method; the character similarity weighting includes: when the similarity of the replaced character is higher than a set similarity threshold, the cost of the replacement operation is the difference between a first set value and the similarity; otherwise, the cost of the replacement operation is a second set value, and the second set value indicates that a penalty value higher than the difference between the first set value and the similarity is assigned to the cost of the replacement operation; the weighted edit distance method is used for the two strings to be matched, and during the replacement operation, the similarity between the two characters to be replaced in the two strings to be matched is queried based on the glyph similarity matrix, and the cost of the replacement operation is weighted by character similarity to obtain the result of the weighted edit distance; the result of the weighted edit distance is normalized to obtain the similarity of the two strings to be matched; A similarity matching module is used to determine a matching result based on the relationship between the similarity of the two character strings to be matched and a preset value; The weighted edit distance method is used for the two strings to be matched, specifically including: Create a (m+1)×(n+1) matrix; Where m is the length of the first string to be matched, n is the length of the second string to be matched, the row index i of the matrix corresponds to the number of characters from left to right in the first string, the column index j of the matrix corresponds to the number of characters from left to right in the second string, and each matrix element D[i][j] in the matrix represents the minimum edit distance among the edit distances required to convert the first i characters of the first string to the first j characters of the second string; Fill the matrix: Among them, D[i-1][j]+1 is the deletion operation, D[i][j-1]+1 is the insertion operation, D[i-1][j-1]+cost is the replacement operation, and cost is the cost of the replacement operation; Query the character similarity in the similarity matrix between the character to be replaced in the first string and the character used as a replacement reference in the second string, and weight the cost by character similarity: Among them, if represents the condition, similarity represents the character similarity, and threshold represents the set similarity threshold; After filling, the value of the matrix element D[m][n] in the matrix represents the weighted edit distance between the first character string and the second character string.

Images