English placename proofreading method considering multi-dimensional character features
By constructing a standardized English place name dataset and using a similarity algorithm based on local character order, this paper addresses the problem that existing place name verification methods fail to effectively utilize multi-dimensional character features, thereby improving the accuracy and efficiency of English place name verification and enhancing the performance of geographic information services.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- YANGZHOU UNIV
- Filing Date
- 2022-11-16
- Publication Date
- 2026-05-22
AI Technical Summary
Existing methods for proofreading English place names fail to effectively utilize the multidimensional character features of place names, resulting in limited accuracy and efficiency of place name data in geographic information services. This is especially true in big data environments, where it is difficult to efficiently identify and correct errors in place names, both in fact and factual.
A standardized English place name dataset was constructed, an index structure based on character statistical features was designed, and a similarity algorithm based on local character order was adopted. English place names were proofread through coarse matching and fine matching, and multi-dimensional character features were used to improve the accuracy and efficiency of proofreading.
It improved the accuracy and efficiency of English place name proofreading, provided more accurate place name input, and improved the overall performance of geographic information services.
Smart Images
Figure CN115759055B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a text proofreading method for English place names, belonging to the field of text-oriented natural language processing technology. Background Technology
[0002] Currently, against the backdrop of the rapid development of strategic emerging industries such as big data, the Internet of Things, and cloud computing, the integration of information technology and geographic information technology is constantly being strengthened, resulting in the emergence of new geographic information service formats, including positioning and navigation, location recommendation, and route planning. Place names are the proprietary names given to geographic entities with specific orientations and geographical ranges, and belong to a basic geographic information resource. Since place names are often used by people to describe spatial locations in daily life, they can play the role of a "human-machine" location information communication bridge in geographic information services. However, as a geographic information service in the era of big data, place name data faces huge challenges in practical applications, mainly in the following three aspects: (1) There are many types of geographic information services supported by place name data resources, which will face many complex application scenarios. Due to inaccurate input, unfamiliarity with users, or non-standard spelling, errors in place name grammar will occur, which will seriously affect the accuracy of each link in subsequent geographic information services; (2) Place name Romanization is a way to provide a Roman alphabet spelling form for place names in countries that use non-Roman alphabet languages, and it is one of the important contents of international standardization of place names. The English alphabet is currently the most widely used version of the Roman alphabet in the world, and English place names have a wide range of applications in international exchanges; (3) The increasing scale of place name data accumulation puts forward higher requirements for geographic information services in terms of operational efficiency and accuracy.
[0003] Text proofreading is a crucial step in natural language processing, enabling the timely detection and correction of word errors in text. As a global language, English spelling proofreading and grammar correction techniques were researched in the 1960s. English text errors mainly fall into three categories: (1) non-word errors, where the words in the text do not actually exist, including substitution, transposition, omission, and insertion errors; (2) true word errors, also known as "contextual errors," which refer to words that actually exist despite the presence of incorrect characters in the text; and (3) grammatical errors, caused by inappropriate word selection. Faced with these different error types, English proofreading primarily employs rule-based methods and classifier-based models. For example, a non-standard word proofreading method based on finite state automata is proposed for non-standard words such as numbers and abbreviations in English text. A Tri-gram grammatical similarity test strategy is used to correct spelling errors. Felice et al. proposed a rule- and phrase-based statistical machine translation (SMT) model system.
[0004] Currently, English text proofreading primarily targets online news, instant messaging, and social media texts, focusing mainly on spelling errors and non-standard expressions such as special characters. There are few existing proofreading methods specifically for place names; most directly adopt or borrow general text proofreading methods, ignoring the inherent expressive patterns and character characteristics of place names. How to effectively utilize the various character characteristics of English place names during proofreading, how to identify inaccurate or incomplete English place names, and how to efficiently provide similar candidate place names for correction have become the key to substantially improving proofreading performance. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a method for English place name proofreading that takes into account multi-dimensional character features. This method provides a basis for error identification and correction of English place names, improving the accuracy and efficiency of English place name proofreading. Furthermore, this method facilitates more accurate English place name input for other types of geographic information services, further improving the overall performance of geographic information services.
[0006] The technical solution adopted by this invention to solve its technical problem is: a method for proofreading English place names that takes into account multi-dimensional character features, the method comprising the following steps:
[0007] Step 1: Constructing a Standardized English Place Name Dataset. A standardized English place name dataset serves as a reference and benchmark for English place name proofreading. It requires collecting English place names from multiple sources and standardizing them to create a dataset with uniform format and accurate content. Specifically, this includes:
[0008] Step 11: Determine the methods for collecting English place name data. Methods for collecting English place name data include place name survey results, geographic mapping results, place name books, and open-source internet maps. One or more of these methods can be used to collect English place name data.
[0009] Step 12: Determining the Scope of English Place Name Data Collection. When collecting English place name data from different sources, it is necessary to follow the temporal, spatial, and typological characteristics of the English place names to be proofread. Specifically, temporal characteristics mean that the time range of the collected English place name data must at least cover the time range of the English place names to be proofread; spatial characteristics mean that the spatial range of the collected English place name data must at least cover the spatial range of the English place names to be proofread; and typological characteristics mean that the types of English place name data collected must at least cover the types of the English place names to be proofread.
[0010] Step 13: Collect, clean, and integrate English place name data. The collected English place name data needs to undergo data cleaning steps such as deduplication, case-sensitivity matching, and removal of special characters. This will integrate the collected English place name data into a format that is standardized, accurate, and descriptively accurate.
[0011] Step 14: Standardize the generation of the English place name dataset. Save the cleaned and integrated English place name data as an English place name dataset in file format.
[0012] Step 2: Generating an English place name index based on character statistical features. In a big data environment, the scale of standardized English place name datasets is usually very large, requiring a suitable index structure to improve the efficiency and accuracy of standardized English place name queries during the proofreading process. Specifically, this includes:
[0013] Step 21: Design of the English Place Name Index Structure. The overall framework of the English place name index structure is based on the basic inverted index model. The index includes an index dictionary file and an index directory file. The index dictionary file stores all English place name entries in the standardized English place name dataset; the index directory file stores the multidimensional character features of each English place name entry and its storage location within the index dictionary file.
[0014] Step 22: Generate the English place name index dictionary file. The index dictionary file stores all English place name entries from the standardized English place name dataset. All English place name entries are stored sequentially without line breaks or gaps, and the index dictionary file has the .bin extension.
[0015] Step 23: Generate the English place name index directory file. The index directory file stores English place name index entries; each English place name entry in the index dictionary file corresponds to an index entry in the index directory file. The character features of each English place name entry in the index dictionary file are statistically analyzed to generate the index entry for that entry. The character features to be analyzed include the number of entries, word details, the first character of each word, and storage location information.
[0016] The number of entries is recorded in the index dictionary file, which contains the number of entries that have the same word details and first character information as the English place name entries corresponding to the current index.
[0017] The word details record the number of characters in the English place name entry corresponding to the current index and the score of that entry, as shown in the scoring instructions in Table 1. The scores of all English characters in the English place name are added together to obtain the score for that English place name entry.
[0018] The word first character information records the number of words in the English place name entry corresponding to the current index, as well as the first character code of the first 4 words in the entry;
[0019] The storage location information records the storage location of the English place name entry corresponding to the current index in the index dictionary file.
[0020] The character feature information obtained from the above statistics is arranged in the order of word count, word details, first character of word, and storage location to obtain an index entry in the index directory file. An English place name index entry is created for each English place name entry in the index dictionary file and stored as an index directory file in .txt format.
[0021] Table 1. Scoring Explanation for Each Character in English Place Names
[0022]
[0023]
[0024] Step 3: Calculate the similarity of English place names based on local character order. This includes an English place name similarity algorithm, used to determine the degree of matching between different English place names during the English place name proofreading process. Specifically, it includes:
[0025] Step 31: Local Order Relationship Types Between Characters. Define the character order types that exist between English place name strings, dividing them into four types: identical, replacement, missing, and added.
[0026] Step 32: Count the number of characters with the same order. Count the number of characters that appear in both English place name strings.
[0027] Step 33: Sequential Character Replacement Count. Count the number of characters that change between characters in the same order in two English place name strings.
[0028] Step 34: Count the number of missing characters in two English place name strings that are present in the longer string but not in the shorter string.
[0029] Step 35: Local character count. Count the number of characters in the two English place name strings that exist in the shorter string but not in the longer string.
[0030] Step 36: Calculate the local character order similarity. Calculate the similarity score between the two English place names using the following formula.
[0031]
[0032] Where, sim ep This is the similarity score for English place names. char2 is a string of English place names with a large number of characters. len(char2) is the length of the char2 string. same indicates characters with the same order. change indicates characters to be replaced in order. reduce indicates missing characters in some places. plus indicates adding characters in some places.
[0033] Step 4: English place name verification based on character matching. Based on a standardized English place name dataset, English place name verification is performed using an English place name index structure based on character statistical features and an English place name similarity algorithm based on local character order, following these steps.
[0034] Step 41: Coarse matching stage, which involves traversing and searching for relevant place names in the English place name index. From the English place name index, we traverse and search for standardized English place names that are related to the English place name to be checked. We then form a candidate English place name set C1 from the found standardized place names, which will be used as the objects for place name search in the subsequent fine matching stage.
[0035] Step 42: Fine-matching stage, using a similarity algorithm to precisely find relevant place name entries. Based on the calculation results of the character local order similarity algorithm, the standard entry with the highest similarity in the candidate English place name set C1 is found. If its similarity is greater than 0.7, the English place name is returned as the proofreading result of this method. Otherwise, the proofreading result of the English place name to be proofread is not returned.
[0036] Beneficial effects:
[0037] 1. This invention provides a basis for error identification and correction of English place names, and has a good proofreading effect on both non-true word errors and true word errors in English place names in the field of natural language processing.
[0038] 2. This invention constructs an index structure suitable for English place names, which can improve the running efficiency of English place name proofreading methods, is applicable to the current big data environment, and provides an English place name character similarity algorithm, making the proofreading results more accurate and reasonable, and improving the accuracy of English place name proofreading.
[0039] 3. This invention is beneficial for providing more accurate English place name input for other types of geographic information services, further improving the overall performance of geographic information services. Attached Figure Description
[0040] Figure 1 This is a flowchart of the method of the present invention.
[0041] Figure 2 This is a schematic diagram of the English place name index dictionary file structure of this invention.
[0042] Figure 3 This is a schematic diagram of the English place name index directory file structure of this invention.
[0043] Figure 4 This is a flowchart of the similarity algorithm based on local character order in this invention.
[0044] Figure 5 Examples of characters with the same order in this invention.
[0045] Figure 6 This is an example of sequential character replacement according to the present invention.
[0046] Figure 7 Example of adding characters to a portion of the present invention.
[0047] Figure 8 This is an example of partially missing characters in this invention. Detailed Implementation
[0048] The invention will now be described in further detail with reference to the accompanying drawings.
[0049] Example 1
[0050] like Figure 1 As shown, this invention provides a method for proofreading English place names that takes into account multi-dimensional character features. The method includes:
[0051] (1) Based on the time range, spatial region and place name type of the English place names to be proofread, collect English place name data from multiple sources and construct a standardized English place name dataset; (2) Extract statistical features for each place name entry in the standardized English place name dataset and construct an English place name index based on this; (3) Based on the local order relationship between characters in English place names, including English place name similarity algorithm; (4) Based on the English place name index and English place name similarity algorithm, design an English place name proofreading process based on character matching, including two stages: coarse matching and fine matching.
[0052] The process and principles of the above-mentioned method for proofreading English place names will be explained in detail below.
[0053] (I) Standardizing the Construction of English Place Name Datasets
[0054] Standardized English place names serve as the benchmark for verification. Based on the characteristics of the English place names to be verified, a standardized English place name dataset needs to be constructed first. The sources of English place name data are diverse, mainly including: place name census results, geographic survey results, place name books, and open-source Internet resources. English place name data is collected through multiple sources, and the collection criteria include the following three aspects: (1) the time range of the English place names to be verified; (2) the spatial region of the English place names to be verified; and (3) the place name type of the English place names to be verified. The collected English place name data needs to cover at least or exceed the above three requirements. For example, if the English place name to be verified is a road name in New York City, USA in 2021, then the time range of collected English place names should at least cover 2021, the spatial region should at least cover the administrative division of New York City, USA, and the place name type should at least include urban roads. After deduplication, case unification, and removal of special characters, the collected multi-source English place name data is saved as a standardized English place name dataset.
[0055] (II) Generation of English place name indexes based on character statistical features
[0056] Step 21: Index File Structure Design. In this method, the English place name index file consists of an index dictionary file and an index directory file. The index dictionary file stores all English place name entries in the standardized English place name dataset, while the index directory file stores the multidimensional character features of each English place name entry and its storage location within the index dictionary file.
[0057] Step 22: Index dictionary file data structure. For example... Figure 2 As shown, the index dictionary file stores all the English place name entries in the standardized English place name dataset, and the storage rule is that all the English place name entries are stored sequentially without line breaks or gaps.
[0058] Step 23: Index directory file data structure. For example... Figure 3 As shown, the index directory file is indexed based on the English place name entries in the index dictionary file. Each English place name in the index dictionary file corresponds to one index in the index directory file. Each index in the index directory file records four parts: the number of entries, word details, the first character of the word, and the storage location. The specific content is as follows:
[0059] Entry count information. Records the number of entries in the entire index dictionary file that share the same word details and first character information as the English place name entry corresponding to the current index.
[0060] Detailed information on the word. Record the number of characters in the English place name entry corresponding to the current index and the score for that entry. The scoring method basically follows the stroke count of each English character, as shown in the scoring instructions in Table 1. The scores of all English characters in the English place name are added together; the sum is the score for that English place name entry.
[0061] First character information for each word. Records the number of words in the English place name entry corresponding to the current index, as well as the first character codes of the first 4 words in the entry. Character codes are case-insensitive and numbered sequentially according to the alphabet, with AZ increasing from 0 to 25. The first character of fewer than 4 words is numbered 26, and all other special characters are numbered 27.
[0062] Storage location information. Records the storage location of the English place name entry corresponding to the current index in the index dictionary file. Since the index dictionary file contains English place names with the same word detail information and first character information, these place name indexes will be merged into a single entry. Therefore, the storage location information will map multiple storage locations in the index dictionary file, with the number of storage locations matching the number of entries in the entry count information. The storage location information will first record the file storage location of the first English place name entry in the index dictionary file corresponding to the current index, and then sequentially store the storage location offsets of other English place name entries.
[0063] Step 24: Arrange the extracted information in the order of word count, word details, first character of word, and storage location. Convert the arranged information into a byte array to obtain an index entry in the index directory file. Repeat step 23 to create an English place name index for each English place name entry in the index dictionary file and store it in the index entry of the index directory file.
[0064] (III) Similarity Algorithm Based on Local Character Order
[0065] like Figure 4 As shown, the character local order-based similarity algorithm compares the character order of two English place name strings cyclically to extract the number of identical, replaced, missing, and added characters. Based on features such as character types and string length, including a similarity calculation formula, it is used to measure the similarity between English place name strings.
[0066] Step 31: Define the character types that exist between the English place name strings, categorizing them into four types: identical characters, character substitutions, partial omissions, and partial additions. Identical characters are those that appear in both strings; substitutions are characters that are transformed between two identical characters; added characters are those that exist in the shorter string but not in the longer string; and missing characters are those that exist in the longer string but not in the shorter string.
[0067] Step 32: For the two English place name strings, compare their lengths and assign the shorter string to char1 and the longer string to char2.
[0068] Step 33: Loop through and compare char1 and char2, extract the common characters, and mark them as "same". In each loop, the comparison starts at the first unmarked character in char1, and the difference in position between the characters identified as "same" in char1 and char2 is no greater than 3; otherwise, they cannot be identified as "same".
[0069] Step 34: Construct a match[] array with a length equal to the length of char2. If match[i] is the character "same", then match[i] contains the position information of "same" in char1; otherwise, match[i] is 0.
[0070] Step 35: Iterate through the `match[]` array. If `match[i]` is 0, increment `reduce` (missing character) by 1 and `len1` by 1. If `match[i]` is not 0, increment `same` by 1, and calculate `len2` using the formula `match[i] - 1` (the first non-zero value in the `match[]` array). Compare the smaller of `len1` and `len2` and add it to the `change` (replacement character) value. Here, `reduce` represents the missing character, `same` represents the same character, and `change` represents the replacement character.
[0071] Step 36: Obtain the change and reduce values;
[0072] Step 37: Calculate the plus value, where plus indicates adding a character;
[0073] Step 38: Calculate the similarity score of English place names using the following formula (sim). ep The calculation is performed, where len(char2) is the length of the char2 string.
[0074]
[0075] (iv) Proofreading of English place names based on character matching
[0076] Step 41: Coarse matching stage, which involves traversing and searching for relevant place name entries in the English place name index;
[0077] The process involves iterating through the English place name index to find standardized English place names that are related to the English place names to be proofread, and then forming a candidate English place name set C1 from these standardized place names. The specific steps include:
[0078] a) Character feature extraction of the English place name to be proofread. The following character features in the place name to be proofread are calculated: number of characters, number of words, scoring information, and first character information. Specifically, the number of characters is the total number of characters in the English place name to be proofread; the number of words is the total number of words in the English place name to be proofread; the scoring information is determined according to the mapping table in Table 1, and the calculation method is the same as in step 23(b); the first character information requires extracting the first character information of the first 4 words in the place name to be proofread, and the calculation method is the same as in step 23(c).
[0079] b) Related Place Name Search Based on English Place Name Index. Using the character features extracted from the English place names to be verified, related English place names are searched from the index file. Let the number of characters o, the number of words p, the rating information q, and the first character information r of the English place name P to be matched be the same as those in the index file. Let the number of characters w, the number of words p, the rating information q, and the first character information be the same as those of the term W in the index file. The search threshold is set to k. w k x k y With k z When the differences between w, x, y, z and o, p, q, r belong to k w k x k y With k z In this case, it can be assumed that W may be related to P, and English place names that meet this characteristic can be marked as c. n .
[0080] c) Form a set of candidate English place names. Iterate through all the found related English place names... n The candidate English place names are collected and formed into a set C1, which will be used as the objects for place name search in the subsequent fine matching stage.
[0081] Step 42: Fine matching stage, based on similarity algorithm to accurately find relevant place name entries;
[0082] Based on the calculation results of the character local order similarity algorithm, the standard term with the highest similarity in the candidate English place name set C1 is found and returned as the proofreading result. Specific steps include:
[0083] a) Calculation of similarity between candidate English place names. Based on the similarity algorithm based on local character order, the similarity between the English place name to be checked and the entries in the candidate English place name set C1 is calculated sequentially, and the entries in C1 are sorted according to the similarity.
[0084] b) Finding the proofreading result. Accurately find the English place name with the highest similarity in the candidate English place name set C1. If its similarity is greater than 0.7, return that English place name as the proofreading result of this method. Otherwise, do not recommend a proofreading result for the English place name to be proofread.
[0085] Example 2
[0086] This invention provides a method for proofreading English place names that takes into account multi-dimensional character features. The method includes:
[0087] (I) Standardizing the Construction of English Place Name Datasets
[0088] This example uses the Global Geographic Name Server (GNS) website as the source of standardized English place name data. It crawls some US place name data from GNS to construct a standardized English place name dataset, which contains 103,954 place name data entries, as an implementation example.
[0089] (II) Generation of English place name indexes based on character statistical features
[0090] Step 21: Index File Structure Design. In this method, the English place name index file consists of an index dictionary file and an index directory file. The index dictionary file stores all English place name entries in the standardized English place name dataset, while the index directory file stores the multidimensional character features of each English place name entry and its storage location within the index dictionary file.
[0091] Step 22: Index dictionary file data structure. The index dictionary file (bin file) stores all English place name entries in the standardized English place name dataset. The storage rule is that all English place name entries are stored sequentially without line breaks or gaps.
[0092] For example: "...Layton Road Bayton Lane Layton Road Layton Avenue Layton RoadLayton Lane Layton Road Layton Rise..." are the four English place names of "Layton Road Bayton Lane", "Layton Road Layton Avenue", "Layton Road Layton Lane" and "Layton Road LaytonRise" respectively.
[0093] Step 23: Index Directory File Data Structure. The index directory file (txt file) is indexed based on the English place name entries in the index dictionary file. Each English place name in the index dictionary file corresponds to one index in the index directory file. Each index in the index directory file records four parts: the number of entries, word details, the first character of the word, and the storage location. Taking the English place name "Lawford Avenue" as an example, the specific content is as follows:
[0094] Entry count information. Records the number of entries in the entire bin file that share the same word details and first character information as the English place name entry corresponding to the current index. For example, if there are 11 English place names in the bin file that share the same word details and first character information as "Lawford Avenue", then the entry count information is marked as "11".
[0095] Detailed word information. This records the number of characters in the English place name entry corresponding to the current index and the score for that entry. The scoring method basically follows the stroke count of each English character, as explained in Table 1. The scores of all English characters in the place name are added together, and the sum is the score for that English place name entry. For example, "Lawford Avenue" has 13 characters and a score of 30, so the detailed word information is marked as "13,30".
[0096] First character information for each word. Records the number of words in the current index corresponding to the English place name entry, as well as the first character codes of the first 4 words in the entry. Character codes are case-insensitive and numbered sequentially alphabetically, with AZ increasing from 0 to 25. Words with fewer than 4 words have a first character code of 26, and all other special characters are numbered 27. For example, "Lawford Avenue" has 2 words and a first character code of 11002626, so the first character information for that word is marked as "2,11002626".
[0097] Storage location information. Records the storage location of the English place name entry corresponding to the current index in the bin file. Since there are English place names in the bin file with the same word detail information and word first character information, the indexes of these place names will be merged into a single entry. Therefore, the storage location information will map multiple storage locations in the bin file, and the number of storage locations is the same as the number of entries in the word count information. The storage location information will first record the file storage location of the first English place name entry in the bin file corresponding to the current index, and then store the storage location offsets of other English place name entries in sequence. English place names with the same word detail information and word first character information as "Lawford Avenue" include "Lawflat Avenue", "Lawling Avenue", etc., so the storage location information needs to record the start and end positions of each English place name stored in the bin file, for example: "1253,23,28,…". Here, 23 is the offset relative to 1253, and 28 is the offset relative to 1276 (note: 1253+23=1276).
[0098] Step 24: Arrange the extracted information in the order of word count, word details, first character of word, and storage location. Convert the arranged information into a byte array to obtain an index record in the index txt file. Repeat step 23 to create an English place name index for each English place name entry in the bin file and store it in the index record of the txt file. The index record for "Lawford Avenue" can be represented as "11; 13, 30; 2, 11002626; 1253, 23, 28, ...".
[0099] (III) Similarity Algorithm Based on Local Character Order
[0100] A character-based local order similarity algorithm iteratively compares the character order of two English place name strings to extract the number of identical, replaced, missing, and added characters. Based on features such as character types and string length, including a similarity calculation formula, it is used to measure the similarity between English place name strings.
[0101] Step 31: Define the character types that exist between English place name strings, which are divided into four categories: identical, replaced, missing, and added.
[0102] Taking the two strings "abcd" and "aebed" as examples, before extracting the relevant characters, the strings are assigned numbers: the shorter one is string 1, and the longer one is string 2.
[0103] like Figure 5 As shown, characters with the same order are characters that exist in both strings. For example, the characters "a", "b", and "d" in string 1 have the same characters in string 2, and can be considered as identical characters.
[0104] like Figure 6 As shown, sequential replacement characters are characters that change between two identical characters. For example, in string 1, the identical characters "b" and "d" are replaced by "c", and in string 2, the same identical characters are replaced by "e". "e" is identified as the replacement character.
[0105] like Figure 7 As shown, locally added characters are characters that exist in the shorter string but not in the longer string. For example, if string 1 contains the character "c" but not string 2, this can be considered an added character.
[0106] like Figure 8 As shown, a partially missing character is a character that exists in the longer string but not in the shorter string. For example, string 1 does not contain the character "e", but "e" exists in string 2, so it is considered a missing character.
[0107] Step 32: For two English place name strings, compare their lengths, assigning the shorter string to char1 and the longer string to char2. If the string lengths are the same, assign the English place name to be checked to char1, and the rest to char2.
[0108] Step 33: Loop through and compare char1 and char2, extract the common characters, and mark them as "same". In each loop, the comparison starts at the first unmarked character in char1, and the difference in position between the characters identified as "same" in char1 and char2 is no greater than 3; otherwise, they cannot be identified as "same".
[0109] Step 34: Construct a match[] array with a length equal to the length of char2. If match[i] is the character "same", then match[i] contains the position information of "same" in char1; otherwise, match[i] is 0.
[0110] Step 35: Iterate through the `match[]` array. If `match[i]` is 0, increment `reduce` (missing character) by 1 and `len1` by 1. If `match[i]` is not 0, increment `same` by 1, and calculate `len2` using the formula `match[i] - 1` (the first non-zero value in the `match[]` array). Compare the smaller of `len1` and `len2` and add it to the `change` (replacement character) value. Here, `reduce` represents the missing character, `same` represents the same character, and `change` represents the replacement character.
[0111] Step 36: Obtain the change and reduce values;
[0112] Step 37: Calculate the plus value, where plus indicates adding a character;
[0113] Step 38: Calculate the similarity score of English place names using the following formula (sim). ep The calculation is performed, where len(char2) is the length of the char2 string.
[0114]
[0115] Taking the English place name "Lawfards Avnue" to be proofread as an example, calculate its similarity to the standard English place name "LawfordAvenue". The algorithm comparison shows that the similarity to the character "same" is 11, the similarity to the missing character "reduce" is 1, the similarity to the replaced character "change" is 1, and the similarity to the added character "plus" is 1. The length of the string "char2" is 13, therefore sim ep = (11×2) / (13×2+1+1+1)≈0.76.
[0116] (iv) Proofreading of English place names based on character matching
[0117] Step 41: Coarse matching stage, which involves traversing and searching for relevant place name entries in the English place name index;
[0118] The process involves iterating through the English place name index to find standardized English place names that are related to the English place name to be proofread, and then forming a candidate English place name set from the found standardized place names. Specific steps include:
[0119] Character feature extraction of the English place name to be proofread. The following character features in the place name to be proofread are statistically analyzed: number of characters, number of words, rating information, and first character information. The number of characters is the total number of characters in the English place name to be proofread; the number of words is the total number of words in the English place name to be proofread; the rating information is determined according to the mapping table in Table 1, and the calculation method is the same as in step 23(b); the first character information requires extracting the first character information of the first 4 words in the place name to be proofread, and the calculation method is the same as in step 23(c). For example, for the English place name to be proofread, “Lawfards Avnue”, the number of characters is 13, the number of words is 2, the rating information is 31, and the first character information is 11002626.
[0120] This method uses an English place name index to find related place names. It leverages character features extracted from the English place names to be checked to search for associated English place names in the index file. Let the number of characters *o*, the number of words *p*, the rating information *q*, and the first character information *r* of the English place name *P* to be matched be the same as those in the index file. Let the number of characters *w*, the number of words *p*, the rating information *q*, and the first character information *z* of the term *W* in the index file be the same as those in the index file. The search threshold is set to *k*. w k x k y With k z When the differences between w, x, y, z and o, p, q, r belong to k w k x k y With k z In this case, it can be assumed that W may be related to P, and English place names that meet this characteristic can be marked as c. n You can set k. w ≤6、k x ≤3、k y ≤15 and k z If two consecutive digits are the same, "Lawford Avenue" will be marked as c1 because the statistical characteristics of the English place name "Lawford Avenue" in the index meet these thresholds.
[0121] Form a candidate English place name set. Iterate through all the found related English place names c. nThe candidate English place names are collected and formed into a set C, which will be used as the objects for place name search in the subsequent fine matching stage. In this embodiment, C = {...; Lawford Avenue; Lawford Lane; Lawford Place; Lawford Road; ...}, which are English place names that meet the threshold range after being traversed and searched, and together they constitute the candidate English place name set.
[0122] Step 42: Fine matching stage, precise search of relevant place name entries based on similarity algorithm;
[0123] Based on the similarity algorithm based on local character order, the standard term with the highest similarity in the candidate English place name set C1 is found and returned as the proofreading result. Specific steps include:
[0124] Similarity calculation of candidate English place names. Based on the similarity algorithm based on local character order, the similarity between the English place name to be checked and the entries in the candidate English place name set C1 is calculated sequentially, and the entries in C1 are sorted according to the similarity.
[0125] The proofreading result is retrieved. The algorithm precisely searches for the English place name with the highest similarity score in the candidate English place name set C1. If the similarity score is greater than 0.7, this English place name is returned as the proofreading result. Otherwise, no proofreading result is recommended. In this example, the place name with the highest similarity score in the candidate English place name set C is "LawfordAvenue," and the similarity score of 0.76 is greater than 0.7. Therefore, the proofreading result for the English place name "Lawfards Avnue" is "Lawford Avenue."
[0126] For those skilled in the art, the device of the present invention is not limited to the exemplary embodiments described above. Any substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and concept of the present invention, should be covered within the protection scope of the present invention.
Claims
1. A method for proofreading English place names that takes into account multi-dimensional character features, characterized in that, The method includes the following steps: Step 1: Standardize the construction of the English place name dataset; The standardized English place name dataset serves as a reference and benchmark for English place name proofreading. It collects English place names from multiple sources and processes them in a standardized manner to form a collection of English place name data with uniform format and accurate content. Step 2: Generate an English place name index based on character statistical features; In a big data environment, the standard English place name dataset is usually very large, requiring an index structure to improve the efficiency and accuracy of standard English place name queries during the proofreading process; Step 21: Design of English place name index structure. Based on the basic pattern of inverted index, the overall framework of English place name index structure is included. The English place name index contains an index dictionary file and an index directory file. The index dictionary file is used to store all English place name entries in the standardized English place name dataset, and the index directory file is used to store the multi-dimensional character features of English place name entries and the storage location of the entry in the index dictionary file. Step 22: Generate the English place name index dictionary file. The index dictionary file stores all the English place name entries in the standardized English place name dataset. All the English place name entries are stored sequentially without line breaks or gaps. The index dictionary file has the suffix .bin. Step 23: Generate the English place name index directory file. The index directory file stores English place name index entries. Each English place name entry in the index dictionary file corresponds to an index entry in the index directory file. Statistically analyze the character features of each English place name entry in the index dictionary file to generate the index entry for that English place name entry. The character features to be analyzed include the number of entries, word details, the first character of the word, and the storage location. Step 3: Calculate the similarity of English place names based on the local order of characters; It includes an English place name similarity algorithm, which is used as a basis for judging the degree of matching between different English place names during the English place name proofreading process; Step 31: Local order relationship types between characters. Define the character order types between English place name strings, and divide them into four types: same, replacement, missing, and added. Step 32: Count the number of characters with the same order in the two English place name strings. Step 33: Sequential replacement character statistics, count the number of characters that change between characters with the same order in the two English place name strings; Step 34: Local missing character statistics, count the number of characters in the two English place name strings that exist in the longer string but not in the shorter string; Step 35: Add local character statistics to count the number of characters in the two English place name strings that exist in the shorter string but not in the longer string; Step 36: Calculate the local character order similarity. Use the following formula to calculate the similarity score between the two English place names: Where, sim ep This is the similarity score for English place names. char2 is a string of English place names with a large number of characters. len(char2) is the length of the char2 string. same indicates characters with the same order. change indicates characters to be replaced in order. reduce indicates missing characters in some places. plus indicates adding characters in some places. Step 4: English place name verification based on character matching. Based on a standardized English place name dataset, using an English place name index structure based on character statistical features and an English place name similarity algorithm based on local character order, English place name verification is performed according to the following steps: Step 41: Coarse matching stage, which involves traversing and searching for relevant place name entries in the English place name index; From the English place name index, we traverse and find the standard English place names that are related to the English place names to be checked. We then form a candidate English place name set C1 from the found standard place names, which will be used as the objects for place name search in the subsequent fine matching stage. Step 42: Fine matching stage, based on similarity algorithm to accurately find relevant place name entries; Based on the calculation results of the character local order similarity algorithm, the standard term with the highest similarity in the candidate English place name set C1 is found. If its similarity is greater than 0.7, the English place name is returned as the proofreading result of this method; otherwise, the proofreading result of the English place name to be proofread is not returned.
2. The method for proofreading English place names that takes into account multi-dimensional character features according to claim 1, characterized in that, Step 1 includes: Step 11: Determine the methods for collecting English place name data. The methods for collecting English place name data include place name survey results, geographic surveying and mapping results, place name books, and open source Internet maps. Collect English place name data from one or more of these methods. Step 12: Determine the scope of English place name data collection. When collecting English place name data from different sources, it is necessary to follow the temporal, spatial, and typological characteristics of the English place names to be verified. Specifically, temporal characteristics mean that the time range of collecting English place name data must at least cover the time range of the English place names to be verified; spatial characteristics mean that the spatial range of collecting English place name data must at least cover the spatial range of the English place names to be verified; and typological characteristics mean that the types of English place name data collected must at least cover the types of the English place names to be verified. Step 13: Collect, clean, and integrate English place name data. For the collected English place name data, it is necessary to perform data cleaning steps such as deduplication, standardization of character capitalization, and deletion of special characters. The collected English place name data will be integrated into a form with uniform format, accurate content, and standardized description. Step 14: Standardize the generation of the English place name dataset. Save the cleaned and integrated English place name data as an English place name dataset in file format.
3. The method for proofreading English place names that takes into account multi-dimensional character features according to claim 1, characterized in that, Step 2 includes: recording the number of entries in the index dictionary file, which is the number of entries that have the same word details and word first character information as the English place name entries corresponding to the current index; The word details record the number of characters in the English place name entry corresponding to the current index and the score result of the entry. Referring to the scoring instructions in Table 1, the scores of all English characters in the English place name are added together, and the sum is the score result of the English place name entry. The word first character information records the number of words in the English place name entry corresponding to the current index, as well as the first character code of the first 4 words in the entry; The storage location information records the storage location of the English place name entry corresponding to the current index in the index dictionary file; The character feature information obtained from the above statistics is arranged in the order of word count, word details, first character of word, and storage location to obtain an index entry in the index directory file. An English place name index entry is created for each English place name entry in the index dictionary file and stored in the form of a txt file as the index directory file. Table 1. Scoring Explanation for Each Character in English Place Names 。