Efficient and robust city information model non-spatial data matching method and system

Abstract

The invention relates to the technical field of data processing, in particular to an efficient and robust city information model non-spatial data matching method and system, which translates each key address element into an independent coding token through a word coding technology, processes by using a Jaccard algorithm in combination with semantic analysis and by taking a word group as a unit, and obtains a non-spatial data matching result. According to the method, semantic features in non-spatial address data can be better captured, meanwhile, in a fuzzy matching stage, a Jaccard algorithm is combined to introduce an edit distance algorithm, step-by-step gradual optimization is achieved, and a proper algorithm is dynamically selected in different steps according to requirements, so that the operation efficiency of the whole algorithm is remarkably improved, the time and calculation cost is reduced, and the method is suitable for large-scale popularization and application. And the accuracy and precision of the matching process are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an efficient and robust urban information model non-spatial data matching method and system. Background Art

[0002] In the application of geographic information systems (GIS) and location-based services, converting non-spatial data (such as text descriptions of addresses) into spatial data (such as longitude and latitude coordinates) and matching them with the spatial database of place names and addresses is the core technology for achieving precise positioning and spatial analysis.

[0003] However, the existing traditional string fuzzy matching algorithms mainly rely on character-level similarity calculations. These algorithms have certain effects when processing simple text matching, but when processing place name and address data involving complex semantics and structures, they often ignore semantic information and phonetic structure, resulting in high computational complexity. In particular, the computational efficiency is low when facing large-scale data sets, making it difficult to meet the efficient processing requirements in practical applications.

[0004] It should be noted that the information disclosed in this background technology section is only intended to increase the understanding of the overall background of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes prior art already known to a person skilled in the art. Summary of the invention

[0005] In order to solve the technical problem that the above conventional matching algorithm cannot meet the efficient processing requirements, the present invention provides an efficient and robust city information model non-spatial data matching method, which comprises the following steps: Acquiring non-spatial address data and preprocessing the non-spatial address data to obtain preprocessed non-spatial address data; Extract and classify the preprocessed non-spatial address data to obtain address data with complete elements and address data with fuzzy elements; For address data with complete elements, a spatialized place name and address database is introduced for accurate comparison and matching; For address data with fuzzy elements, a spatialized place name address database is introduced, and fuzzy matching is performed using the Jaccard algorithm and the minimum edit distance algorithm. According to the matching results, the latitude and longitude coordinates and related spatial information of the corresponding spatial address data are returned.

[0006] Furthermore, the preprocessing of non-spatial address data at least includes correcting coding anomalies, eliminating noise words, removing redundant addresses, separating alternative addresses, replacing special characters, eliminating redundant spaces, removing meaningless punctuation, processing reduplication or repeated information, and keyword filling or word replacement.

[0007] Furthermore, the extraction and classification of elements of the preprocessed non-spatial address data comprises the following steps: Analyze the semantic features in the preprocessed non-spatial address data and accurately extract the key geographic elements in the address; According to the extracted key geographic elements, the preprocessed non-spatial address data are classified into address data with complete elements and address data with fuzzy elements.

[0008] Furthermore, the key geographical elements include at least provincial, municipal, district and street levels.

[0009] Furthermore, the step of introducing the address data with complete elements into the spatialized place name and address database for accurate comparison and matching comprises the following steps: According to the extracted key geographic elements, the spatialized place name and address database is traversed to match and search for the corresponding spatial address data in the spatialized place name and address database: If the match is successful, the latitude and longitude coordinates of the corresponding spatial address data and related spatial information are returned; If the matching fails, the corresponding address data with complete elements will be classified as address data with ambiguous elements.

[0010] Furthermore, the address data with fuzzy elements is introduced into the spatialized place name address database, and fuzzy matching is performed using the Jaccard algorithm and the minimum edit distance algorithm: Based on the extracted key geographic elements, a candidate set is extracted from the spatialized place name and address database by using a Jaccard algorithm; Based on the extracted candidate set, the similarity between the element-fuzzy address data and all the spatial address data in the candidate set is calculated using the minimum edit distance algorithm, and the latitude and longitude coordinates and related spatial information of the spatial address data with the highest similarity in the candidate set are returned according to the calculation results.

[0011] Furthermore, extracting the candidate set comprises the following steps: Based on the extracted key geographical elements, the word encoding technology is used to encode and translate them into tokens, each token represents a key geographical element; The Jaccard algorithm is used to calculate the similarity between the element-fuzzy address data and each spatial address data in the spatialized place name address database. ; Based on similarity The calculation results of the spatialized place name address database and the similarity of the address data with element fuzziness are selected Spatialized address data exceeding a certain threshold is taken as a candidate set.

[0012] Furthermore, the similarity between the element-blurred address data and each spatial address data in the spatialized place name address database is Expressed as:

[0013] Among them, A is the token formed by translating the address data with blurred elements, and B is the token formed by translating the spatial address data in the spatialized place name and address database.

[0014] Furthermore, the calculation of the similarity between the element-blurred address data and all the spatial address data in the candidate set by the minimum edit distance algorithm comprises the following steps: The minimum edit distance algorithm is used to calculate the minimum number of edits to convert the fuzzy address data into the address data of each space in the candidate set. ; Based on minimum number of edits Calculate the similarity between the fuzzy address data and each spatial address data in the candidate set , then the similarity Expressed as:

[0015] in, is the minimum number of edits, The maximum string length of the spatialized address data in the candidate set.

[0016] Furthermore, the present invention also provides an efficient and robust city information model non-spatial data matching system, which uses any of the above-mentioned efficient and robust city information model non-spatial data matching methods to perform data matching on non-spatial data.

[0017] Based on the above, the present invention provides an efficient and robust urban information model non-spatial data matching method and system. Compared with the prior art, each key address element is translated into an independent coding token through word encoding technology, and the Jaccard algorithm is combined with semantic analysis to process it in phrases, which can better capture the semantic features in non-spatial address data. At the same time, the edit distance algorithm is introduced in combination with the Jaccard algorithm in the fuzzy matching stage, realizing step-by-step optimization, and dynamically selecting suitable algorithms according to needs in different steps, thereby significantly improving the operating efficiency of the overall algorithm, reducing time and computing costs, and improving the accuracy and precision of the matching process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. The positional relationship described in the drawings in the following description is based on the direction of the components in the drawings unless otherwise specified.

[0019] Figure 1 A schematic diagram of the overall process of an efficient and robust urban information model non-spatial data matching method provided by an embodiment of the present invention; Figure 2 A schematic diagram of a process for extracting and classifying elements of pre-processed non-spatial address data provided by an embodiment of the present invention; Figure 3 A schematic diagram of a process for introducing address data with complete elements into a spatialized place name and address database for accurate comparison and matching provided by an embodiment of the present invention; Figure 4 A schematic diagram of a process for introducing address data with fuzzy elements into a spatialized place name and address database for fuzzy matching provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In the description of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more. In addition, the term "including" and any variation thereof all mean "at least including".

[0022] In order to solve the technical problem that the above-mentioned conventional matching algorithm cannot meet the requirements of efficient processing, or to achieve at least one of the above-mentioned advantages or other advantages, the present invention provides an efficient and robust urban information model non-spatial data matching method and system to improve the efficiency, accuracy and precision of the matching process.

[0023] The following is a detailed description with reference to specific embodiments and drawings.

[0024] Embodiment 1 See also Figure 1 This embodiment provides an efficient and robust urban information model non-spatial data matching method, which includes the following steps: Acquiring non-spatial address data and preprocessing the non-spatial address data to obtain preprocessed non-spatial address data; Extract and classify the preprocessed non-spatial address data to obtain address data with complete elements and address data with fuzzy elements; For address data with complete elements, a spatialized place name and address database is introduced for accurate comparison and matching; For address data with fuzzy elements, a spatialized place name address database is introduced, and fuzzy matching is performed using the Jaccard algorithm and the minimum edit distance algorithm. According to the matching results, the latitude and longitude coordinates and related spatial information of the corresponding spatial address data are returned.

[0025] In specific implementation, the sources of non-spatial address data are very diverse, and they usually come not only from official data, but also from business data and open source data, etc. These data from different sources often follow different standards and formats, which leads to huge challenges in the subsequent matching process. Therefore, it is necessary to pre-process the acquired non-spatial address data.

[0026] The main task of the process of preprocessing the acquired non-spatial address data is to ensure the cleanliness and standardization of the address data, which at least includes correcting coding anomalies, eliminating noise words, removing redundant addresses, separating alternative addresses, replacing special characters, eliminating extra spaces, removing meaningless punctuation, processing reduplication or repeated information, keyword filling or word replacement, so as to facilitate the subsequent spatial processing of the non-spatial address data.

[0027] For example, coding exception modification can check and correct abnormal characters caused by coding problems in non-spatial address data, convert all non-uniformly coded data into a unified format, and ensure that there is no garbled code. Eliminating noise words can remove irrelevant noise words in non-spatial address data to ensure data cleanliness and improve efficiency and accuracy. Eliminating redundant addresses can remove duplicate address records to ensure the uniqueness of non-spatial address data. Separating alternative addresses can separate addresses separated by separators and perform subsequent matching processes separately. Replacing special characters can replace special characters in non-spatial address data that may affect matching with standard characters. Eliminating redundant spaces can clean up redundant spaces in non-spatial address data to ensure data compactness and standardization. Eliminating meaningless punctuation can remove unnecessary punctuation in non-spatial address data to simplify data. Processing reduplication or repeated information can delete or merge repeated words or information in non-spatial address data. Keyword filling or word replacement can supplement keywords or replace words according to business needs to ensure semantic integrity.

[0028] After the preprocessing of the non-spatial address data is completed, the preprocessed non-spatial address data needs to be spatialized. The core of this step is to extract and classify the elements of the preprocessed non-spatial address data to obtain address data with complete elements and address data with fuzzy elements, which is convenient for the subsequent matching process.

[0029] Address data with complete elements can be directly introduced into the spatialized place name and address database for accurate matching. Based on the extracted key geographic elements, they are directly compared with the spatialized place name and address database to find the corresponding spatialized place name and address. In this process, hierarchical indexing and database query technology can be used to find the corresponding place name and address text information in the spatialized place name and address database based on the extracted key geographic elements, and return its latitude and longitude coordinates and related spatial information.

[0030] After the address data with fuzzy elements are introduced into the spatialized place name address database, fuzzy matching is performed using the Jaccard algorithm and the minimum edit distance algorithm. This step can calculate the similarity between the spatialized address data in the spatialized place name address database based on the extracted key geographic elements, and return the latitude and longitude coordinates and related spatial information of the corresponding spatialized address data based on the calculation results.

[0031] It should be noted that in the technical solution of this embodiment, the spatialized place name and address database can be collected from multiple sources, such as official data, business data, open source place name and address data, etc., which will not be elaborated here.

[0032] In some preferred embodiments, Figure 2 As shown, extracting and classifying the elements of the preprocessed non-spatial address data includes the following steps: Analyze the semantic features in the preprocessed non-spatial address data and accurately extract the key geographic elements in the address; According to the extracted key geographic elements, the preprocessed non-spatial address data are classified into address data with complete elements and address data with fuzzy elements.

[0033] Specifically, the semantic features of the preprocessed non-spatial address data are parsed, and the address text of the preprocessed non-spatial address data is decomposed into key geographic elements with different semantic levels. Each key geographic element represents a specific geographic level in the non-spatial address data, including at least provincial, municipal, district, street, community, road name, road number, unit, room, other remaining elements, etc. Optionally, regular expressions, mapping tables, natural language models, etc. can be used to extract key geographic elements.

[0034] Furthermore, the classification based on key geographic elements is accurate to at least the road number level. That is, when the key geographic elements extracted from a non-spatial address data have multiple levels such as provincial, municipal, district, street, community, road name, and road number, it can be classified as address data with complete elements. On the contrary, if the extracted key geographic elements are incomplete, missing, or ambiguous, or there are non-spatial address data with key geographic elements that do not belong to the above levels, it is classified as address data with ambiguous elements.

[0035] On the basis of the above, if Figure 3 As shown, the following steps are included to introduce address data with complete elements into the spatialized place name address database for accurate comparison and matching: According to the extracted key geographic elements, the spatialized place name and address database is traversed to match and search for the corresponding spatial address data in the spatialized place name and address database: If the match is successful, the latitude and longitude coordinates of the corresponding spatial address data and related spatial information are returned; If the matching fails, the corresponding address data with complete elements will be classified as address data with ambiguous elements.

[0036] First, based on the extracted key geographic elements, the spatialized place name and address database is traversed to match and find the spatial address corresponding to each key geographic element. Then, the matching results of each key geographic element are combined to find a common subset. If the match is successful, the latitude and longitude coordinates and related spatial information of the subset are returned to complete the precise match.

[0037] If the match fails, it means that there may be inconsistencies in database standards or errors in extracting key geographic elements. The corresponding address data with complete elements will be reclassified as address data with fuzzy elements for fuzzy matching.

[0038] Furthermore, if Figure 4 As shown in the figure, the following steps are included for introducing address data with fuzzy elements into the spatialized place name address database for fuzzy matching: Based on the extracted key geographic elements, the candidate set is extracted from the spatialized place name address database using the Jaccard algorithm; Based on the extracted candidate set, the similarity between the element-fuzzy address data and all the spatial address data in the candidate set is calculated using the minimum edit distance algorithm, and the latitude and longitude coordinates and related spatial information of the spatial address data with the highest similarity in the candidate set are returned according to the calculation results.

[0039] In the specific implementation, at the initial stage of fuzzy matching, firstly, based on the extracted key geographical elements, the word encoding technology is used to encode and translate to form tokens, and each token represents a key geographical element. The influence of the number of characters between words on the text weight of the overall address data is eliminated, and the subsequent matching process is carried out in phrases, which can better capture the semantic features in the spatial address with fuzzy elements and greatly improve the accuracy and precision of the matching process.

[0040] Secondly, the inverted index and Jaccard algorithm are used to calculate the similarity between the address data with blurred elements and each spatial address data in the spatialized place name address database. According to the similarity The calculation results of the spatialized place name address database and the similarity of the address data with element fuzziness are selected The spatialized address data exceeding a certain threshold is used as a candidate set. Specifically, the threshold range can be defined according to actual needs.

[0041] An inverted index is a data structure that allows documents containing these words to be quickly located based on the words in the text. This index structure is different from the traditional forward index, which is text-centric, while the inverted index is word-centric, mapping each word to a list of texts containing it. The implementation of the inverted index is based on a basic concept: rather than determining the word by the text, the location of the text is determined by the word. The construction of an inverted index involves mapping each address association in a text collection to each unique word that appears in it. This data structure is particularly suitable for quickly locating documents containing specific keywords in a large text collection. This index can greatly reduce the complexity of the algorithm and improve retrieval efficiency.

[0042] The Jaccard algorithm is a method to measure the similarity between two texts. It measures the similarity by calculating the ratio of the intersection and union of two sets. Then the similarity between the element fuzzy address data and each spatial address data in the spatialized place name address database is Expressed as:

[0043] Among them, A is the token formed by translating the address data with blurred elements, and B is the token formed by translating the spatial address data in the spatialized place name and address database.

[0044] Finally, the minimum edit distance algorithm is used to calculate the minimum number of edits required to convert the fuzzy address data into the address data of each space in the candidate set. . Based on the minimum number of edits Calculate the similarity between the fuzzy address data and each spatial address data in the candidate set .

[0045] The similarity between the fuzzy address data of the element and the spatial address data in the candidate set is Expressed as:

[0046] in, is the minimum number of edits, The maximum string length of the spatialized address data in the candidate set.

[0047] This formula normalizes the minimum edit distance to the range of 0 to 1, and obtains a similarity value between 0 and 1. The closer this value is to 1, the greater the similarity between the element-fuzzy address data and the spatialized address data in the candidate set.

[0048] The minimum edit distance algorithm is a method for measuring the similarity between two strings. It defines the similarity by calculating the minimum number of edit operations required to convert one string into another. These edit operations include inserting, deleting, or replacing characters. By building a matrix to store the distances calculated step by step, the minimum edit distance between the two strings is finally obtained. In this way, the similarity between the address data string with ambiguous elements and the spatialized address data string in the candidate set is measured, and based on the calculation result of the similarity, the closest spatialized address data is selected, and its corresponding spatial coordinates and place name address data are returned.

[0049] However, when processing long strings, the computational complexity of the minimum edit distance algorithm is O(m*n), where m and n are the lengths of the two strings respectively. When the string becomes longer, the computational time will increase significantly, resulting in high computational complexity. At the same time, for large-scale text or data sets, calculating the minimum edit distance one by one may cause performance bottlenecks. Therefore, a single minimum edit distance algorithm cannot meet the needs of efficient processing.

[0050] In some preferred embodiments, the present invention also provides an efficient and robust city information model non-spatial data matching system, which uses any of the above-mentioned efficient and robust city information model non-spatial data matching methods to perform data matching on non-spatial data.

[0051] In summary, compared with the prior art, the efficient and robust urban information model non-spatial data matching method and system provided by the present invention translates each key address element into an independent coding token through word encoding technology, and uses the Jaccard algorithm combined with semantic analysis to process it in phrases, which can better capture the semantic features in non-spatial address data. At the same time, the edit distance algorithm is introduced in combination with the Jaccard algorithm in the fuzzy matching stage, and step-by-step optimization is achieved. In different steps, the appropriate algorithm is dynamically selected according to the needs, thereby significantly improving the operating efficiency of the overall algorithm, reducing time and computing costs, and improving the efficiency, accuracy and precision of the matching process.

[0052] Although the terms such as complete elements and element fuzziness are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

[0053] In addition, those skilled in the art should understand that, although there are many problems in the prior art, each embodiment or technical solution of the present invention can be improved in only one or several aspects, without having to solve all the technical problems listed in the prior art or background technology at the same time. Those skilled in the art should understand that the content not mentioned in a claim should not be used as a limitation on the claim.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient and robust urban information model non-spatial data matching method, characterized by: include: Acquiring non-spatial address data and preprocessing the non-spatial address data to obtain preprocessed non-spatial address data; Extract and classify the preprocessed non-spatial address data to obtain address data with complete elements and address data with fuzzy elements; For address data with complete elements, a spatialized place name and address database is introduced for accurate comparison and matching; For address data with fuzzy elements, a spatialized place name address database is introduced, and fuzzy matching is performed using the Jaccard algorithm and the minimum edit distance algorithm. According to the matching results, the latitude and longitude coordinates and related spatial information of the corresponding spatial address data are returned.

2. The efficient and robust urban information model non-spatial data matching method according to claim 1 is characterized by: The preprocessing of non-spatial address data at least includes correcting coding anomalies, eliminating noise words, removing redundant addresses, separating alternative addresses, replacing special characters, eliminating redundant spaces, removing meaningless punctuation marks, processing reduplication or repeated information, and keyword filling or word replacement.

3. The efficient and robust urban information model non-spatial data matching method according to claim 1 is characterized by: The element extraction and classification of the preprocessed non-spatial address data comprises the following steps: Analyze the semantic features in the preprocessed non-spatial address data and accurately extract the key geographic elements in the address; According to the extracted key geographic elements, the preprocessed non-spatial address data are classified into address data with complete elements and address data with fuzzy elements.

4. The efficient and robust urban information model non-spatial data matching method according to claim 3 is characterized by: The key geographical elements include at least provincial, municipal, district and street levels.

5. The efficient and robust urban information model non-spatial data matching method according to claim 1 is characterized by: The step of introducing the address data with all elements into the spatialized place name address database for accurate comparison and matching comprises the following steps: According to the extracted key geographic elements, the spatialized place name and address database is traversed to match and search for the corresponding spatial address data in the spatialized place name and address database: If the match is successful, the latitude and longitude coordinates of the corresponding spatial address data and related spatial information are returned; If the matching fails, the corresponding address data with complete elements will be classified as address data with ambiguous elements.

6. The efficient and robust urban information model non-spatial data matching method according to claim 1 is characterized by: The address data with fuzzy elements is introduced into the spatialized place name address database, and fuzzy matching is performed using the Jaccard algorithm and the minimum edit distance algorithm: Based on the extracted key geographic elements, a candidate set is extracted from the spatialized place name and address database by using a Jaccard algorithm; Based on the extracted candidate set, the similarity between the element-fuzzy address data and all the spatial address data in the candidate set is calculated using the minimum edit distance algorithm, and the latitude and longitude coordinates and related spatial information of the spatial address data with the highest similarity in the candidate set are returned according to the calculation results.

7. The efficient and robust urban information model non-spatial data matching method according to claim 6 is characterized by: The extracting candidate set comprises the following steps: Based on the extracted key geographical elements, the word encoding technology is used to encode and translate them into tokens, each token represents a key geographical element; The Jaccard algorithm is used to calculate the similarity between the element-fuzzy address data and each spatial address data in the spatialized place name address database. ; Based on similarity The calculation results of the spatialized place name address database and the similarity of the address data with element fuzziness are selected Spatialized address data exceeding a certain threshold is taken as a candidate set.

8. The efficient and robust urban information model non-spatial data matching method according to claim 7 is characterized by: The similarity between the element-fuzzy address data and each spatial address data in the spatialized place name address database Expressed as: Among them, A is the token formed by translating the address data with blurred elements, and B is the token formed by translating the spatial address data in the spatialized place name and address database.

9. The efficient and robust urban information model non-spatial data matching method according to claim 6 is characterized by: The method of calculating the similarity between the element-fuzzy address data and all spatial address data in the candidate set by using the minimum edit distance algorithm comprises the following steps: The minimum edit distance algorithm is used to calculate the minimum number of edits to convert the fuzzy address data into the address data of each space in the candidate set. ; Based on minimum number of edits Calculate the similarity between the fuzzy address data and each spatial address data in the candidate set , then the similarity Expressed as: in, is the minimum number of edits, The maximum string length of the spatialized address data in the candidate set.

10. An efficient and robust urban information model non-spatial data matching system, characterized by: The non-spatial data is matched using the efficient and robust urban information model non-spatial data matching method as described in any one of claims 1 to 9.

Images