Population standard address matching method and system based on analytic data warehouse

Abstract

The present disclosure provides a population standard address matching method and system based on an analytical data warehouse, and relates to the technical field of standard address matching, which comprises the following steps: collecting and sorting a StarRocks data warehouse containing standard addresses; obtaining external population data and preprocessing the external population data; matching the preprocessed population data information with the population address information in the StarRocks data warehouse containing standard addresses; adopting a multi-level address matching algorithm to associate the areas matched to the standard addresses, and storing the areas not matched to the standard addresses in a special address library; updating and maintaining the address data in the special address library, and incorporating the address data in the special address library into the StarRocks data warehouse containing standard addresses after dynamic updating; and the present disclosure has high accuracy and consistency in address matching.

Description

Technical Field

[0001] This disclosure relates to the field of standard address matching technology, specifically to a population standard address matching method and system based on analytical data warehouses. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Standard address matching technology is a method that matches and compares an input address with addresses in a standard address database, aiming to improve the accuracy and consistency of address data.

[0004] Existing standard address matching methods include data preparation, address resolution, data matching, matching degree evaluation, and output of address matching results. However, existing address matching methods have certain limitations, specifically including the following shortcomings:

[0005] 1) Data quality limitations: Standard address matching relies on an accurate and complete standard address database. If there are incorrect, outdated, or missing address data in the database, it will affect the accuracy of the matching results.

[0006] 2) Complex address processing: Some addresses have complex structures or special formats, such as large commercial buildings and public institutions. These addresses may not be effectively processed by traditional standard address matching techniques.

[0007] 3) Place name polysemy: In some cases, a place name may have multiple meanings; for example, the same place name may exist in different geographical locations. This may lead to ambiguous or incorrect results in standard address matching.

[0008] 4) Data Updates and Maintenance: The standard address database requires regular updates and maintenance to reflect new address changes, road alterations, new buildings, etc. Failure to update the database in a timely manner will result in inaccurate or outdated matching results.

[0009] 5) Limitations of matching algorithms: Different matching algorithms are suitable for different scenarios and data conditions. Choosing an appropriate matching algorithm may require experimentation and adjustment to obtain the best matching results.

[0010] 6) Accuracy of geographic data: Standard address matching may be limited by the accuracy of geographic data. If the geographic data is inaccurate or contains errors, it will affect the accuracy of the matching results. Summary of the Invention

[0011] To address the aforementioned issues, this disclosure proposes a population standard address matching method and system based on an analytical data warehouse. It employs the StarRocks high-performance analytical data warehouse, which utilizes technologies such as vectorization, MPP architecture, CBO, intelligent materialized views, and a real-time updatable columnar storage engine to achieve multi-dimensional, real-time, and high-concurrency data analysis, thereby enabling standard address matching.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] Population standard address matching methods based on analytical data warehouses include:

[0014] Collect and organize a StarRocks data warehouse containing standard addresses;

[0015] Acquire external population data and preprocess the external population data;

[0016] The preprocessed population data is matched with the population address information in the StarRocks data warehouse containing standard addresses. A multi-level address matching algorithm is used to associate the areas that match the standard addresses and store the areas that do not match the standard addresses in a special address database. The address data in the special address database is updated and maintained, and then incorporated into the StarRocks data warehouse containing standard addresses after dynamic updates.

[0017] The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] The database initialization module is used to collect and organize the StarRocks data warehouse containing standard addresses;

[0020] The data acquisition module is used to acquire external population data and preprocess the external population data.

[0021] The address matching module is used to match preprocessed population data with population address information in the StarRocks data warehouse containing standard addresses. It uses a multi-level address matching algorithm to associate areas that match standard addresses, store areas that do not match standard addresses in a special address database, update and maintain the address data in the special address database, and incorporate them into the StarRocks data warehouse containing standard addresses after dynamic updates.

[0022] The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the population standard address matching method based on an analytical data warehouse.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform the population standard address matching method based on an analytical data warehouse.

[0027] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0028] In terms of data quality, the method disclosed herein improves the accuracy and consistency of address matching by establishing and maintaining a high-quality standard address database and employing a highly accurate matching algorithm.

[0029] Regarding matching efficiency, this disclosure improves address matching efficiency and reduces the time consumption of the matching process by optimizing the matching algorithm and data structure.

[0030] This disclosure improves the data update mechanism by enhancing the data update and maintenance mechanism to ensure that the standard address database reflects new address changes and modifications in a timely manner, providing more accurate matching results.

[0031] In terms of expanding application areas, this disclosure, through improved and innovative standard address matching technology, can expand its application areas to include logistics management, geographic information systems, location services, etc., providing broader and more accurate address-related services. Attached Figure Description

[0032] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0033] Figure 1This is a conventional standard method matching process according to embodiments of this disclosure;

[0034] Figure 2 This is a flowchart illustrating the population standard address matching method based on an analytical data warehouse, according to an embodiment of this disclosure. Detailed Implementation

[0035] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0037] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0038] Example 1

[0039] One embodiment of this disclosure provides a population standard address matching method based on an analytical data warehouse, the steps of which include:

[0040] Step 1: Collect and organize the StarRocks data warehouse containing standard addresses;

[0041] Step 2: Obtain external population data and preprocess the external population data;

[0042] Step 3: Match the preprocessed population data with the population address information in the StarRocks data warehouse containing standard addresses. Use a multi-level address matching algorithm to associate the areas that match the standard addresses, store the areas that do not match the standard addresses in a special address database, update and maintain the address data in the special address database, and incorporate it into the StarRocks data warehouse containing standard addresses after dynamic updates.

[0043] The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data.

[0044] As one embodiment, the specific implementation process of the population standard address matching method based on analytical data warehouse disclosed herein is as follows:

[0045] Step 1: Collect and organize the StarRocks data warehouse containing standard addresses;

[0046] Step 2: Obtain external population data and preprocess the external population data;

[0047] Step 3: Match the preprocessed population data with the population address information in the StarRocks data warehouse containing standard addresses. Use a multi-level address matching algorithm to associate the areas that match the standard addresses, store the areas that do not match the standard addresses in a special address database, update and maintain the address data in the special address database, and incorporate it into the StarRocks data warehouse containing standard addresses after dynamic updates.

[0048] The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data.

[0049] As one embodiment, in step 1, the data warehouse organized by this disclosure is a StarRocks data warehouse containing standard addresses, and the process of establishing this data warehouse is as follows:

[0050] Address information is collected via an address collection terminal and then stored in the StarRocks data warehouse through a message middleware. The address information table uses geocoding rules to uniquely encode each address, and a unique hash index is created. This geocoding ensures that each address is stored only once in the database, avoiding data redundancy and duplication. The unique hash index also improves the speed of address information retrieval, thereby enhancing database performance.

[0051] The database coverage is as follows: Standard address matching techniques typically rely on standard address databases, which contain verified and standardized address data. This disclosure utilizes the StarRocks high-performance analytical data warehouse. StarRocks is a high-performance analytical data warehouse that uses vectorization, MPP architecture, CBO, intelligent materialized views, and a real-time updatable columnar storage engine to achieve multidimensional, real-time, and high-concurrency data analysis. StarRocks supports efficient data import from various real-time and offline data sources and direct analysis of data in various formats on data lakes. StarRocks is compatible with the MySQL protocol and can be integrated with MySQL clients and common BI tools. StarRocks also features horizontal scalability, high availability, high reliability, and ease of maintenance. It is widely used in real-time data warehouses, OLAP reporting, and data lake analytics scenarios.

[0052] Furthermore, in step 2, the external population data includes the input address, username, and authentication information.

[0053] In step 2, external population data is collected. This data can come from various sources, such as government agencies, census bureaus, and statistics bureaus. The data may be in the form of tables, documents, databases, etc.

[0054] Next, the collected external population data is preprocessed. Preprocessing includes the following steps:

[0055] Data cleaning involves checking for missing values, outliers, and duplicates in the data and processing them accordingly. For example, missing values ​​can be removed, outliers can be filled with the mean or median, and duplicates can be eliminated.

[0056] Data transformation: Converting data into a format suitable for analysis. For example, converting categorical data into numerical data, or string data into numerical data, etc.

[0057] Data normalization: Converting data into a unified standard format. For example, converting all numerical data into decimal form, or converting all date data into a specific format.

[0058] After the above preprocessing, the external population data becomes cleaner, neater, and easier to analyze.

[0059] In step 3: The preprocessed population data information is matched with the population address information in the StarRocks data warehouse containing standard addresses. A multi-level address matching algorithm is used to calculate the similarity between the input address and the standard address.

[0060] Specifically, the automatic parsing of Chinese addresses using the BERT model involves the following steps:

[0061] 1) Data Preparation: First, a large amount of Chinese address data needs to be collected, including different types of addresses such as home addresses, company addresses, and school addresses. Simultaneously, each address needs to be labeled with its corresponding area, street, and house number.

[0062] 2) Data Preprocessing: The collected address data needs to be preprocessed, including word segmentation, stop word removal, and stemming. Simultaneously, the text data needs to be converted to a format that the BERT model can process, i.e., a serialization format.

[0063] 3) Model Construction: Using the BERT model for Chinese address parsing. The BERT model is a pre-trained language model that can automatically learn text representations and is suitable for various natural language processing tasks. One or more classifier layers can be added to the BERT model for address classification and parsing.

[0064] 4) Model Training: The BERT model is trained using the collected address data. During training, address data is input into the model, along with corresponding area, street, and house number information as labels. By optimizing the loss function, the model learns patterns in the address data, thereby improving the accuracy of address resolution.

[0065] 5) Model Evaluation: During training, methods such as cross-validation can be used to evaluate the model's performance. Simultaneously, metrics such as accuracy, recall, and F1 score can be used to measure the model's performance.

[0066] In the matching algorithm and accuracy described, standard address matching technology may employ specific matching algorithms and evaluation metrics to ensure the accuracy and consistency of address matching. This disclosed matching algorithm is multi-level. The first level is based on keyword or regular expression matching algorithms to quickly bind to the town / street / community level. The second level is based on building unit / household number and already matched community regular expressions to the room / household, for example, ( / / .*Huimei Huayueyuan\\D*2\\D*1\\D*101) regular expression matches room 101, unit 1, building 2, Huimei Huayueyuan. Additionally, it supports custom keyword matching strategies and alias matching, such as Donghuayuan actually corresponding to Donghuayuan, Lianchi Living Area actually corresponding to Dormitory 3, and other similar area names, forming a special area dictionary to further improve accuracy.

[0067] The preprocessed population data is matched against population address information in the StarRocks data warehouse containing standard addresses. A multi-level address matching algorithm is used. First, an initial match is performed based on the town / street keyword, then the communities under the town / street are matched. The associated town / street is corrected based on the unique address name of the community. This process continues, matching the sub-communities under the community and recursively correcting the previous level, similar to backpropagation in a neural network. The last level is the household / room, where regular expression matching is used. For example, the regular expression \D*2\\D*1\\D*101) matches room 101, unit 1, building 2, sub-community xxx. Then, the previous level is corrected layer by layer. Areas that match the standard address are associated, while areas that do not match the standard address are stored in a special address database. The address data in the special address database is updated and maintained, and after dynamic updates, it is incorporated into the StarRocks data warehouse containing standard addresses.

[0068] The data update and maintenance method involves regularly updating and maintaining the StarRocks data warehouse containing standard addresses, saving new address changes and modifications, and automatically triggering standard address matching actions whenever a standard address changes. Because standard addresses require manual review after collection, manual review and approval will trigger standard address matching actions. Updating addresses (including aliases, former names, etc.) will also trigger quasi-address matching actions.

[0069] Example 2

[0070] One embodiment of this disclosure provides a population standard address matching system based on an analytical data warehouse, including:

[0071] The database initialization module is used to collect and organize the StarRocks data warehouse containing standard addresses;

[0072] The data acquisition module is used to acquire external population data and preprocess the external population data.

[0073] The address matching module is used to match preprocessed population data with population address information in the StarRocks data warehouse containing standard addresses. It uses a multi-level address matching algorithm to associate areas that match standard addresses, store areas that do not match standard addresses in a special address database, update and maintain the address data in the special address database, and incorporate them into the StarRocks data warehouse containing standard addresses after dynamic updates.

[0074] The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data.

[0075] Example 3

[0076] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the population standard address matching method based on an analytical data warehouse.

[0077] Example 4

[0078] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform the population standard address matching method based on an analytical data warehouse.

[0079] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0081] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A population standard address matching method based on analytical data warehouse, characterized in that, include: Collect and organize a StarRocks data warehouse containing standard addresses; The process of building a StarRocks data warehouse is as follows: Address information is collected through an address collection terminal, and then stored in the StarRocks data warehouse through a message middleware. The address information is uniquely encoded in the address information table using the place name address geocoding rule, and a unique hash index is also established. Acquire external population data and preprocess the external population data; The preprocessed population data is matched with the population address information in the StarRocks data warehouse containing standard addresses. A multi-level address matching algorithm is used to associate the areas that match the standard addresses and store the areas that do not match the standard addresses in a special address database. The address data in the special address database is updated and maintained, and then incorporated into the StarRocks data warehouse containing standard addresses after dynamic updates. The data update and maintenance method is to regularly update and maintain the StarRocks data warehouse containing standard addresses, save new address changes and modifications, and actively trigger standard address matching once the standard address changes. After the standard address is collected, it is manually reviewed. After the manual review is passed, the standard address matching action is triggered. Updating the address will also trigger the standard address matching action. The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data. The multi-level address matching algorithm performs an initial match based on the town / street keywords, then matches the communities under the town / street, and corrects the associated town / street based on the unique address name of the community. This process continues, and when matching the sub-communities under the community, it recursively corrects the previous level. When the last level is the household / room, it uses regular expression matching to correct the previous level layer by layer. The keyword custom matching strategy also supports alias matching, forming a special area dictionary.

2. The population standard address matching method based on an analytical data warehouse as described in claim 1, characterized in that, The StarRocks data warehouse is an analytical data warehouse that uses vectorization, MPP architecture, CBO, intelligent materialized views, and real-time updated columnar storage engine technology to achieve multi-dimensional, real-time, and high-concurrency data analysis.

3. The population standard address matching method based on analytical data warehouse as described in claim 1, characterized in that, StarRocks data warehouse supports efficient data import from various real-time and offline data sources, and also supports direct analysis of data in various formats on the database.

4. The population standard address matching method based on an analytical data warehouse as described in claim 1, characterized in that, StarRocks data warehouse is compatible with the MySQL protocol and can be integrated with common BI tools using MySQL clients.

5. The population standard address matching method based on analytical data warehouse as described in claim 1, characterized in that, The external population data includes the input address, username, and authentication information.

6. The population standard address matching method based on analytical data warehouse as described in claim 1, characterized in that, The preprocessed population data is matched with population address information in the StarRocks data warehouse, which contains standard addresses. This includes using a matching algorithm to calculate the similarity between the input address and the standard address.

7. A population standard address matching system based on an analytical data warehouse, characterized in that, include: The database initialization module is used to collect and organize the StarRocks data warehouse containing standard addresses. The process of building a StarRocks data warehouse is as follows: Address information is collected through an address collection terminal, and then stored in the StarRocks data warehouse through a message middleware. The address information is uniquely encoded in the address information table using the place name address geocoding rule, and a unique hash index is also established. The data acquisition module is used to acquire external population data and preprocess the external population data. The address matching module matches preprocessed population data with population address information in the StarRocks data warehouse containing standard addresses. It employs a multi-level address matching algorithm to associate regions that match standard addresses and store regions that do not match standard addresses in a special address database. The address data in the special address database is updated and maintained, and after dynamic updates, it is incorporated into the StarRocks data warehouse containing standard addresses. The data update and maintenance method involves periodically updating and maintaining the StarRocks data warehouse containing standard addresses, saving new address changes and modifications. A change in a standard address will automatically trigger a standard address matching action. After standard addresses are collected, they undergo manual review; once the manual review is passed, the standard address matching action is triggered. Updating an address will also trigger a standard address matching action. The multi-level address matching algorithm includes: the first level is to bind the standard address to the town / street / community level based on keyword or regular expression matching algorithm; the second level is to match the household room based on the building unit number and the already matched community regular expression matching, thereby realizing the matching of standard addresses of population data. The multi-level address matching algorithm performs an initial match based on the town / street keywords, then matches the communities under the town / street, and corrects the associated town / street based on the unique address name of the community. This process continues, and when matching the sub-communities under the community, it recursively corrects the previous level. When the last level is the household / room, it uses regular expression matching to correct the previous level layer by layer. The keyword custom matching strategy also supports alias matching, forming a special area dictionary.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the population standard address matching method based on an analytical data warehouse as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform the population standard address matching method based on an analytical data warehouse as described in any one of claims 1-6.

Images