Address standardization method, electronic device and medium

Abstract

The present invention provides an address standardization method, electronic device, and medium, which receive address business tables pushed by various business systems, extract relevant data from the address business tables into a personnel-related address intermediate table, and insert the ID card information associated with each address business table into an address ID card processing status table of a database; establish a determined standard address index and a pending standard address index in a search engine; initialize a copy of standard address information, and pull all collected standard address information that meets standard specifications into the determined standard address index as an initial address; and standardize all address information in the personnel-related address intermediate table using a personnel aggregation method based on the address ID card processing status table, the personnel-related address intermediate table, the determined standard address index, and the pending standard address index. The advantages of the present invention are that it can improve the accuracy of address standardization and has better applicability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an address standardization method, electronic equipment, and medium. Background Art

[0002] With the rapid development of information technology, various platforms have collected and accumulated a large amount of address data. However, this address data is often non-standard and non-uniform due to factors such as human input, diverse data sources, and regional differences. For example, the same address description may be diverse, inaccurate, incorrect, with homophones, abbreviations, and other issues.

[0003] The non-standard nature of address data presents numerous difficulties and challenges for its analysis and application. Therefore, standardization of address data is imperative to more effectively utilize these data resources. Currently, common address standardization methods include rule-based address parsing and natural language training. The rule-based address parsing method uses fixed rules to parse and standardize addresses, but its applicability is limited by regional and cultural factors, making it difficult to cover all situations. The natural language training method automatically recognizes and standardizes addresses through training on large amounts of sample data. While this method excels in handling complex situations, its applicability may be limited in areas with insufficient data or in specific fields. Therefore, there is an urgent need for an address standardization method that improves accuracy and applicability. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an address standardization method, electronic equipment and medium, which can improve the accuracy of address standardization and have better applicability.

[0005] In a first aspect, the present invention provides an address standardization method, the method comprising the following steps:

[0006] Receive the address business table pushed by each business system, extract the relevant data in the address business table into the personnel-related address intermediate table, and insert the ID card information associated with each address business table into the address ID card processing status table of the database;

[0007] Establish an index of confirmed standard addresses and an index of pending standard addresses in the search engine; at the same time, initialize a set of standard address information, and pull all collected standard address information that meets the standard specifications into the confirmed standard address index as the initial address;

[0008] According to the address ID card processing status table, the personnel-related address intermediate table, the determined standard address index and the standard address index to be reviewed, all address information in the personnel-related address intermediate table is standardized by using personnel aggregation.

[0009] Furthermore, the extraction of relevant data from the address business table to the personnel-associated address intermediate table is specifically as follows: NiFi is used to extract relevant data from the address business table to the personnel-associated address intermediate table, and the personnel-associated address intermediate table is stored in a search engine. The personnel-associated address intermediate table includes the following fields: information primary key number, ID number, name, contact number, address code, address details, administrative region where the address is located, address type, longitude, latitude, registration time, number of associations, storage time, standardized address or standard address confidence; wherein, the value of the information primary key number is: MD5 (ID number and address details); the value of the address type is: source table.

[0010] Furthermore, the address ID card processing status table includes the following fields: ID number, whether processed, storage time, processing time or priority.

[0011] Furthermore, the determined standard address index and the pending standard address index both include the following fields: information primary key number, standard address, standard address ngram segmentation, standard address ik segmentation, address type, longitude, latitude, province, city, county, township or village, among which the value of the information primary key number is: MD5 (address details).

[0012] Furthermore, the standardization of all address information in the personnel-associated address intermediate table by personnel aggregation based on the address ID card processing status table, the personnel-associated address intermediate table, the determined standard address index, and the pending standard address index specifically includes:

[0013] (1) Use QueryDatabaseTable to pull the data to be processed from the address ID card processing status table, convert the Avro format file pulled by QueryDatabaseTable into JSON array format through the ConvertAvroToJSON component, then use the SplitJson component to convert the data to be processed into independent JSON data files, and use EvaluateJsonPath to extract the ID card attribute in the JSON data file, that is, the document number;

[0014] (2) Using the ID number to call the personnel address standardization interface, the personnel address standardization interface is an HTTP interface, the input parameter is the ID number, and the operation result is returned through the personnel address standardization interface, the operation result is a success flag or a failure flag;

[0015] (3) Obtain the operation result through EvaluateJsonPath and update the operation result to the document number corresponding to the address ID card processing status table.

[0016] Furthermore, the specific processing flow of the personnel address standardization interface is as follows:

[0017] A. Store the complete data of districts, counties, towns, villages and their abbreviations of the entire prefecture-level city into the JDK memory in a map format to obtain an area mapping table. The area mapping table is used as a global parameter of the personnel address standardization interface. The naming format of the area mapping table is prefecture-level city abbreviation + areaMap;

[0018] B. The specific operation method of the personnel address standardization interface is addressHandle(ID number). The specific operation steps are as follows:

[0019] Step B1: Initialize a JSON object obj and set the initial return value as a success flag. At the same time, set a failure flag. If an exception occurs during the execution process, the failure flag is returned.

[0020] Step B2: Initialize an empty string set dzSet, which is used to store addresses that can be used as standard addresses;

[0021] Step B3: query the personnel-associated address intermediate table based on the ID number, and obtain all address arrays associated with the ID number from the personnel-associated address intermediate table, thereby obtaining a JSON array dzList;

[0022] Step B4: Process the JSON array dzList as follows:

[0023] Step B41, obtain the i-th associated address object gldz in the JSON array dzList;

[0024] Step B42: Get the address details field of the associated address object gldz and assign it to str;

[0025] Step B43: Initialize a JSON object maxBzObj and a variable maxBzScore=0, wherein the variable maxBzScore is used to store the maximum similarity, and the JSON object maxBzObj is used to store the standard address information of the maximum similarity;

[0026] Step B44: Loop through other addresses in the JSON array dzList and compare them with the current address details str to find a suitable standard address object maxBzObj, specifically including:

[0027] Step B441, obtain the address details qtdzstr of other current addresses;

[0028] Step B442: Call the similarAddr(str,qtdzstr,dzSet) method to obtain the compared standard address information. The return object bzObj of the similarAddr(str,qtdzstr,dzSet) method includes the standard address and the standard address confidence score bzscore.

[0029] Step B443: If the standard address confidence score bzscore is -1, it means that the current address detail str is not a detailed address, and the other addresses in the JSON array dzList are not traversed.

[0030] If the standard address confidence bzscore is greater than maxBzScore, the standard address object maxBzObj is updated to the return object bzObj of the similarAddr(str,qtdzstr,dzSet) method, and maxBzScore is set to bzscore; if the standard address confidence bzscore is equal to maxBzScore, the length of the standard address object maxBzObj is compared with the length of the standard address in the return object bzObj, and the longer standard address is taken as the standard address of the standard address object maxBzObj. If the lengths are the same, the standard address object maxBzObj is not changed; at the same time, return to step B441 to continue to obtain the address details qtdzstr of the next other address for comparison until all other addresses in the JSON array dzList are traversed;

[0031] Step B45: If maxBzScore is less than or equal to 0.9 or bzscore is -1, the current address details str are standardized according to the region mapping table of the personnel address standardization interface to obtain a temporary standard address. The standard address corresponding to the associated address object gldz is set as the temporary standard address, the standard address confidence bzscore is set to 1, and the temporary standard address is stored in the string set dzSet.

[0032] If maxBzScore is greater than 0.9, the standard address and maxBzScore in the standard address object maxBzObj are written to the standard address and standard address confidence bzscore corresponding to the associated address object gldz, so as to facilitate the subsequent update of the personnel-associated address intermediate table; at the same time, the standard address with a maxBzScore greater than 0.9 is stored in the string set dzSet;

[0033] Step B46: Return to step B41 and continue to obtain the next associated address object gldz for processing until the standard address and standard address confidence bzscore are searched for each associated address object gldz in the JSON array dzList. At this point, each associated address object gldz in the JSON array dzList has two additional attributes: standard address and standard address confidence.

[0034] Step B47: Update the personnel-related address intermediate table according to the attributes of each related address object gldz in the JSON array dzList;

[0035] Step B48, looping to determine whether the standard address string in the string set dzSet is in the determined standard address index, if so, no processing is performed; if not, the standard address string is inserted into the standard address index to be reviewed.

[0036] Furthermore, the calling of the similarAddr(str,qtdzstr,dzSet) method to obtain the compared standard address information specifically includes:

[0037] Step C1: Initialize a JSON object res, which is used to store return parameters, set the initial value of the standard address to the input address string str, and set the standard address confidence bzscore to 0;

[0038] Step C2: Use the HanLP word segmenter to perform word segmentation on the input address string str and another address string qtdzstr, thereby obtaining a word segmentation array;

[0039] Step C3: Based on the word segmentation results, the address string str and the address string qtdzstr are standardized using the regional mapping table of the personnel address standardization interface;

[0040] Step C4: Determine whether the standardized address string str is in the string set dzSet. If so, set the standard address confidence bzscore to 1 and directly return the result.

[0041] If not, determine whether the standardized address string str contains only basic information. If so, set the standard address confidence bzscore to -1 and directly return the result; if not, proceed to step C5;

[0042] Step C5: Continue to perform word segmentation processing on the standardized address string str and the address string qtdzstr, and determine whether there are numerals in the two address strings. If there are numerals, compare whether the numerals in the two address strings are the same. If the numerals are different, set the standard address confidence score bzscore to 0 and directly return the result;

[0043] If the numerals are the same, determine whether the two address strings contain each other. If so, select the longer address string as the standard address, set the standard address confidence bzscore to 1, and directly return the result. If the numerals do not exist or the two address strings do not contain each other, proceed to step C6.

[0044] Step C6: Use the Jaccard similarity algorithm to calculate the similarity value of the two address strings, use the longer address string as the standard address, set the standard address confidence bzscore as the similarity value, and return the result.

[0045] Furthermore, the method also includes: obtaining the standard address to be reviewed from the standard address to be reviewed index, and giving several alternative addresses with the closest similarity to the standard address to be reviewed through the word segmentation field in the determined standard address index, and if it is determined after review that the standard address to be reviewed is the same as one of the alternative addresses, then the corresponding standard address in the personnel-associated address intermediate table is replaced with the alternative address; if it is determined after review that the standard address to be reviewed is different from all the given alternative addresses, then the standard address to be reviewed is inserted into the determined standard address index as the standard address.

[0046] In a second aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0047] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0048] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages: by aggregating the address business tables pushed by various business systems, and extracting the relevant data of each address business table into the personnel-associated address intermediate table, the ID card information associated with each address business table is inserted into the address ID card processing status table of the database; at the same time, a determined standard address index and a pending standard address index are established in the search engine, and according to the address ID card processing status table, the personnel-associated address intermediate table, the determined standard address index and the pending standard address index, all addresses associated with each person in the personnel-associated address intermediate table are compared and standardized by using a personnel aggregation method; by adopting the above-mentioned technical solution, the address business table can be well processed to record relatively concise address information, and regardless of the large or small amount of data, it can be well standardized, thereby improving the accuracy of address standardization and improving applicability.

[0049] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] Figure 1 This is a flowchart of an implementation method of an address standardization method in Embodiment 1 of the present invention;

[0052] Figure 2 This is a schematic structural diagram of an electronic device in a third embodiment of the present invention;

[0053] Figure 3 Schematic diagram of the structure of the medium in the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0054] Example 1

[0055] This embodiment provides an address standardization method, such as Figure 1 As shown, the method includes the following steps:

[0056] Receive the address business table pushed by each business system, extract the relevant data in the address business table into the personnel-related address intermediate table zjb_rydzglxx, and insert the ID card information associated with each address business table into the database's address ID card processing status table zjb.zjb_sfzdz; because there are various different business systems in actual use, and the address data collected by different business systems are often non-standard and in non-uniform formats, it is necessary to extract the relevant data from the address business table pushed by each business system into the personnel-related address intermediate table zjb_rydzglxx to facilitate subsequent standardization of addresses;

[0057] Establish a determined standard address index zjb_bzdz and a pending standard address index zjb_dshdz in the search engine, wherein the search engine may adopt elasticsearch. The determined standard address index zjb_bzdz is used to store addresses that have been determined as standard addresses, and the pending standard address index zjb_dshdz is used to store standard addresses that are pending for review. At the same time, initialize a copy of standard address information, and pull all collected standard address information that meets the standard specifications into the determined standard address index zjb_bzdz as the initial address, so that the standard address in the determined standard address index zjb_bzdz can be used to compare and review other addresses in the future.

[0058] According to the address ID card processing status table zjb.zjb_sfzdz, the personnel-related address intermediate table zjb_rydzglxx, the determined standard address index zjb_bzdz and the standard address index to be reviewed zjb_dshdz, all address information in the personnel-related address intermediate table zjb_rydzglxx is standardized by using personnel aggregation.

[0059] The present invention aggregates the address business tables pushed by various business systems, extracts the relevant data of each address business table into a personnel-related address intermediate table, and inserts the ID card information associated with each address business table into the address ID card processing status table of the database; at the same time, a determined standard address index and a pending standard address index are established in the search engine, and according to the address ID card processing status table, the personnel-related address intermediate table, the determined standard address index and the pending standard address index, all addresses associated with each person in the personnel-related address intermediate table are compared and standardized in a personnel aggregation manner; by adopting the above technical solution, the address business table can well process the relatively concise address information recorded, and can well perform standardized processing regardless of whether the data volume is large or small, thereby improving the accuracy of address standardization and improving applicability.

[0060] In some embodiments of the present invention, extracting the relevant data from the address business table to the personnel-related address intermediate table zjb_rydzglxx is specifically as follows:

[0061] NiFi is used to extract the relevant data in the address business table into the personnel-related address intermediate table zjb_rydzglxx. The personnel-related address intermediate table zjb_rydzglxx is saved in the search engine. The personnel-related address intermediate table zjb_rydzglxx includes the following fields: information primary key number (xxzjbh), ID number (zjhm), name (xm), contact number (lxdh), address code (dzbm), address details (dzxq), administrative division of the address (dzszxzqh), address type (dzlx), longitude (jd), latitude (wd), registration time (djsj), number of associations (glcs), storage time (rksj), standardized address (i.e., standard address bzdz) or standard address confidence (bzscore); wherein, the value of the information primary key number is: MD5 (document number and address details), that is, the information primary key number is obtained by performing an MD5 operation on the document number and address details; the value of the address type is: source table. wherein:

[0062] NiFi is an easy-to-use, powerful and reliable data pulling, data processing and distribution system for automating the flow of data between systems.

[0063] The search engine may adopt elasticsearch. ElasticSearch is a search server based on Lucene, which provides a distributed multi-user full-text search engine based on a RESTful web interface.

[0064] MD5, or Message-Digest Algorithm 5, is used to ensure the integrity and consistency of information transmission. It is one of the hash algorithms widely used in computers. The function of MD5 is to calculate data (such as Chinese characters) into another fixed-length value.

[0065] In some embodiments of the present invention, the address ID card processing status table zjb.zjb_sfzdz includes the following fields: ID number (zjhm), whether processed (status), storage time (rksj), processing time (clsj) or priority (level); wherein, status defaults to 0, and 0 represents unprocessed, status 1 represents processed, and status 2 represents processing failure; level is represented by a number, and the smaller the number, the higher the priority. The higher the priority, the higher the priority will be processed first. For example, the person registered in the household registration table has the highest level. If the data volume has a priority processing order, the level can be used to control the processing flow. As a specific embodiment of the present invention, the address ID card processing status table zjb.zjb_sfzdz can be stored in a greenplum database.

[0066] Since the present invention needs to be standardized by personnel aggregation in the specific implementation, it is necessary to accurately prepare an address ID card processing status table zjb.zjb_sfzdz in advance so that the personnel data can be pulled and processed according to the Nifi process.

[0067] In some embodiments of the present invention, the determined standard address index zjb_bzdz and the pending standard address index zjb_dshdz both include the following fields: information primary key number (xxzjbh), standard address (bzdz), standard address ngram segmentation (bzdz_ngram), standard address ik segmentation (bzdz_ik), address type (dzlx), longitude (jd), latitude (wd), province (province), city (city), county (county), township (township) or village (village), wherein the value of the information primary key number is: MD5 (address details), that is, the address details are MD5-transformed to obtain the information primary key number; the address type includes: 1-standard address table, 2-confirmation in the pending review table. Among them, the standard address ik segmentation specifically refers to the use of ik segmentation to segment the standard address, and the standard address ngram segmentation specifically refers to the use of ngram model to segment the standard address.

[0068] In some embodiments of the present invention, the pulling of all the collected standard address information that meets the standard specifications into the determined standard address index zjb_bzdz as the initial address is specifically: pulling the collected standard address information in the "One Standard and Three Realities" into the determined standard address index zjb_bzdz as the initial address. The "One Standard and Three Realities" specifically refers to the standard address, actual population, actual housing and actual unit. In order to promote grassroots governance and realize the informatization of basic work and the basicization of information work, the relevant units have completed the collection of the "One Standard and Three Realities" data. Compared with other business systems, the standard address in the "One Standard and Three Realities" is composed of elements such as "administrative division + township street + street lane + house number + community (group) + building row number + unit number + household room", and the address is detailed and accurate. Therefore, by directly pulling the standard address information in the "One Standard and Three Realities" into the determined standard address index, the reliability of subsequent comparison and review can be improved.

[0069] In some embodiments of the present invention, the standardization of all address information in the personnel-associated address intermediate table zjb_rydzglxx by personnel aggregation based on the address ID card processing status table zjb.zjb_sfzdz, the personnel-associated address intermediate table zjb_rydzglxx, the determined standard address index zjb_bzdz, and the pending standard address index zjb_dshdz specifically includes:

[0070] (1) Use QueryDatabaseTable to pull the data to be processed in the address ID card processing status table, and convert the Avro format file pulled by QueryDatabaseTable into json array format through the ConvertAvroToJSON component, that is, convert the Avro format into json format to facilitate processing, and then use the SplitJson component to convert the data to be processed into independent json data files one by one, so that the data can be processed one by one, and use EvaluateJsonPath to extract the ID card attribute in the json data file, that is, the document number (zjhm); at the same time, when the present invention is implemented, if the data has a priority processing order, the level can be used to control the processing flow, for example, the query condition plus level=1 is processed first, and the other levels are processed in sequence according to different priorities; wherein, the specific format of the json data file can be as follows:

[0071] {"zjhm":"35052119890218XXXX","status":"0","rksj":"2024-01-01

[0072] 11:11:11","level":1};

[0073] (2) Using the ID number to call the personnel address standardization interface, the personnel address standardization interface is an HTTP interface, the input parameter is the ID number (zjhm), and the operation result is returned through the personnel address standardization interface. The operation result is a success flag or a failure flag. For example, the returned operation result is {"code":"1"}, where code=1 indicates success and code=2 if it fails;

[0074] (3) Obtain the operation result through EvaluateJsonPath, and update the operation result to the document number corresponding to the address ID card processing status table. When the present invention is implemented, it is necessary to update the operation result code to the status of the document number corresponding to the address ID card processing status table, that is, status = code (code = 1 indicates success, code = 2 indicates failure).

[0075] More specifically, the specific processing flow of the personnel address standardization interface is as follows:

[0076] A. Store the complete data of districts, counties, towns (streets), villages (communities), and their abbreviations in the JDK memory in a map format to obtain an area mapping table. This area mapping table is used to assist in the subsequent standardization of addresses. The area mapping table is used as a global parameter of the personnel address standardization interface so that it can be called at any time as needed. The naming format of the area mapping table is the abbreviation of the city + areaMap;

[0077] B. The specific operation method of the personnel address standardization interface is addressHandle(ID number), that is, addressHandle(zjhm). After the ID number (zjhm) is passed in, a JSON string can be returned, such as {"code":"1"}. The specific operation steps are as follows:

[0078] Step B1: Initialize a JSON object obj and set the initial return value to a success flag, i.e., {"code":"1"}, and a failure flag, i.e., {"code":"2"}. If an exception occurs during execution, the failure flag is returned; if successful, the success flag is returned.

[0079] Step B2: Initialize an empty string set dzSet, which is used to store addresses that can be used as standard addresses, so as to facilitate comparison between the standard addresses stored in the string set dzSet and other addresses;

[0080] Step B3: query the personnel-related address intermediate table zjb_rydzglxx according to the ID number (zjhm), obtain all address arrays associated with the ID number (zjhm) from the personnel-related address intermediate table zjb_rydzglxx, and thus obtain a JSON array dzList;

[0081] Step B4: Process the JSON array dzList as follows:

[0082] Step B41: Obtain the i-th associated address object gldz in the JSON array dzList, where i is a positive integer. In a specific implementation of the present invention, it is necessary to cyclically obtain and process the associated address objects gldz in the JSON array dzList until all associated address objects gldz in the JSON array dzList are processed.

[0083] Step B42: Get the address details field of the associated address object gldz and assign it to str, i.e., str = gldz.getStr("dzxq"). In this way, str stores the address details of the i-th associated address object gldz currently retrieved from the JSON array dzList.

[0084] Step B43: Initialize a JSON object maxBzObj and a variable maxBzScore=0, wherein the variable maxBzScore is used to store the maximum similarity, and the JSON object maxBzObj is used to store the standard address information of the maximum similarity;

[0085] Step B44: Loop through other addresses in the JSON array dzList and compare them with the current address details str to find a suitable standard address object maxBzObj, specifically including:

[0086] Step B441, obtain the address details qtdzstr of the current other address, that is, obtain an address other than the current address details str from the JSON array dzList;

[0087] Step B442: Call the similarAddr(str,qtdzstr,dzSet) method to obtain the compared standard address information. The return object bzObj of the similarAddr(str,qtdzstr,dzSet) method includes the standard address and the standard address confidence score bzscore.

[0088] Step B443: If the standard address confidence score bzscore is -1, it means that the current address detail str is not a detailed address, and the other addresses in the JSON array dzList are not traversed.

[0089] If the standard address confidence bzscore is greater than maxBzScore, the standard address object maxBzObj is updated to the return object bzObj of the similarAddr(str,qtdzstr,dzSet) method, and maxBzScore is set to bzscore; if the standard address confidence bzscore is equal to maxBzScore, the lengths of the standard addresses in the standard address object maxBzObj and the return object bzObj are compared, and the longer standard address is taken as the standard address of the standard address object maxBzObj. If the lengths are the same, the standard address object maxBzObj is not changed; at the same time, return to step B441 to continue obtaining the address details qtdzstr of the next other address for comparison until all other addresses in the JSON array dzList are traversed; after performing the above comparison operation, the variable maxBzScore stores the maximum similarity, and the standard address object maxBzObj stores the standard address information with the greatest similarity;

[0090] Step B45: If maxBzScore (i.e., the maximum similarity after comparison) is less than or equal to 0.9 or bzscore is -1, the current address details str are standardized according to the region mapping table of the personnel address standardization interface to obtain a temporary standard address. The standard address corresponding to the associated address object gldz is set as the temporary standard address, the standard address confidence bzscore is set to 1, and the temporary standard address is stored in the string set dzSet.

[0091] If maxBzScore (i.e., the maximum similarity after comparison) is greater than 0.9, the standard address and maxBzScore in the standard address object maxBzObj are written to the standard address and standard address confidence bzscore corresponding to the associated address object gldz, so as to facilitate the subsequent update of the person-associated address intermediate table; at the same time, the standard addresses with maxBzScore greater than 0.9 are stored in the string set dzSet;

[0092] In specific implementation, addresses with maxBzScore greater than 0.9 can basically be considered standard addresses. Therefore, after the comparison of each associated address object gldz is completed, if maxBzScore is greater than 0.9, the standard address and variable maxBzScore in the standard address object maxBzObj are directly written into the standard address and standard address confidence bzscore corresponding to the associated address object gldz. If maxBzScore is less than or equal to 0.9 or bzscore is -1, the current address details str are standardized according to the area mapping table of the personnel address standardization interface to obtain a temporary standard address, and the standard address corresponding to the associated address object gldz is set as the temporary standard address. This standardization method can greatly improve accuracy. Of course, in specific implementation, the present invention does not only take the value 0.9. In actual use, this value can be adjusted according to actual needs.

[0093] Step B46: Return to step B41 and continue to obtain the next associated address object gldz for processing until the standard address and standard address confidence bzscore are searched for each associated address object gldz in the JSON array dzList. At this time, each associated address object gldz in the JSON array dzList has two additional attributes: standard address bzdz and standard address confidence bzscore.

[0094] Step B47: Update the personnel-related address intermediate table according to the attributes of each associated address object gldz in the JSON array dzList, that is, update the standard address bzdz and standard address confidence bzscore corresponding to each associated address in the personnel-related address intermediate table;

[0095] Step B48: Loop to determine whether the standard address string in the string set dzSet is in the determined standard address index. If so, no processing is performed; if not, the standard address string is inserted into the standard address index to be reviewed, so as to facilitate subsequent review of the standard address string, thereby improving the accuracy of the standard address string in the string set dzSet.

[0096] Furthermore, since similarAddr(str,qtdzstr,dzSet) is used to obtain the similarity of two addresses, where str is the address string for obtaining the standard address, qtdzstr is the other address string to be compared with str for address similarity, and dzSet is used to temporarily store the standard address combination that has been identified as the standard address of the person, the calling of the similarAddr(str,qtdzstr,dzSet) method to obtain the compared standard address information specifically includes:

[0097] Step C1: Initialize a JSON object res, which is used to store return parameters, set the initial value of the standard address to the input address string str, and set the standard address confidence bzscore to 0;

[0098] Step C2: Use the HanLP tokenizer to perform token processing on the input address string str and another address string qtdzstr, thereby obtaining a token array. For example, for No. 3, Dandi Lane, District BB, AA City, after tokenization by the HanLP tokenizer, the token array becomes [AA City / District BB / Dandi / Lane / Morpheme 3 / Number / Quantifier].

[0099] Step C3: Based on the word segmentation results, the address string str and the address string qtdzstr are standardized using the regional mapping table of the personnel address standardization interface to complete and standardize the address information as much as possible. Specifically, the completion and standardization are performed according to the format of the regional mapping table, namely, province (autonomous region, municipality) - city (region) - county (municipal district, autonomous county, county-level city) - township (street) - village (community). The main purpose is to use the word segmentation array from front to back as the key to obtain the complete standard corresponding to the regional mapping table, and replace the content in the array according to the most complete registration in the regional mapping table. Among them, after str and qtdzstr are standardized by the HanLP word segmenter and the regional mapping table, the two address strings str1 and qtdzstr can be obtained respectively.

[0100] Step C4: Determine whether the standardized address string str is in the string set dzSet. If so, set the standard address confidence score bzscore to 1, indicating a complete match, and return the result directly. Since the address string str is in the string set dzSet, it means that the address string str has been compared with all addresses before and has been set as the most standard address. Therefore, the address string str can be directly used as the standard address and returned directly.

[0101] If not, determine whether the standardized address string str contains only basic information, that is, only contains basic information such as province, city, and county, without more detailed address information. If so, set the standard address confidence bzscore to -1, indicating that it cannot be compared with other addresses for similarity, and directly return the result; if not, proceed to step C5;

[0102] Step C5: Continue to perform word segmentation processing on the standardized address string str and the address string qtdzstr, and determine whether there are numerals in the two address strings. If there are numerals, compare whether the numerals in the two address strings are the same. If the numerals are different, set the standard address confidence score bzscore to 0, indicating that the two addresses are not similar, and directly return the result;

[0103] If the numerals are the same, determine whether the two address strings contain each other. If so, select the address string with the longer length as the standard address, set the standard address confidence bzscore to 1, and directly return the result. Of course, if the two address strings are the same length, directly use the address string str as the standard address; if there is no numeral or the two address strings do not contain each other, proceed to step C6;

[0104] Step C6: Calculate the similarity between the two address strings using the Jaccard similarity algorithm, use the longer address string as the standard address, set the standard address confidence score bzscore as the similarity value, and return the result. The Jaccard similarity algorithm is an algorithm that can calculate the similarity between two strings to obtain a similarity value.

[0105] It should be noted that, in the specific implementation of the present invention, the return result (ie, JSON object res) of the similarAddr(str, qtdzstr, dzSet) method includes the standard address and the standard address confidence bzscore.

[0106] In some embodiments of the present invention, the method also includes: obtaining the standard address to be reviewed from the standard address to be reviewed index zjb_dshdz, and giving several alternative addresses with the closest similarity to the standard address to be reviewed through the word segmentation field in the determined standard address index zjb_bzdz, and if it is determined after review that the standard address to be reviewed is the same as one of the alternative addresses, then the corresponding standard address in the personnel-associated address intermediate table zjb_rydzglxx is replaced with the alternative address; if it is determined after review that the standard address to be reviewed is different from the given alternative address, then the standard address to be reviewed is inserted into the determined standard address index as the standard address. When the present invention is implemented specifically, it is necessary to manually review the standard address to be reviewed in the standard address to be reviewed index zjb_dshdz; at the same time, it should be noted that after reviewing a standard address to be reviewed in the standard address to be reviewed index zjb_dshdz, the standard address to be reviewed is deleted from the standard address to be reviewed index zjb_dshdz. The present invention can continuously enrich the content of the standard addresses in the determined standard address index by manually reviewing the standard addresses to be reviewed in the standard address index to be reviewed, and inserting the reviewed standard addresses to be reviewed as standard addresses into the determined standard address index, thereby facilitating comparison using the standard addresses in the determined standard address index.

[0107] Based on the same inventive concept, this application provides an electronic device embodiment corresponding to the first embodiment, see the second embodiment for details.

[0108] Example 2

[0109] This embodiment provides an electronic device, such as Figure 2 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0110] Since the electronic device described in this embodiment is the device used to implement the method in Example 1 of this application, based on the method described in Example 1 of this application, those skilled in the art will be able to understand the specific implementation of the electronic device of this embodiment and its various variations. Therefore, how the electronic device implements the method in the embodiment of this application will not be described in detail here. As long as the device used by those skilled in the art to implement the method in the embodiment of this application falls within the scope of protection to be provided by this application.

[0111] Based on the same inventive concept, this application provides a storage medium corresponding to Example 1, see Example 3 for details.

[0112] Example 3

[0113] This embodiment provides a computer-readable storage medium, such as Figure 3 As shown, a computer program is stored thereon, and when the computer program is executed by a processor, any implementation method in Example 1 can be implemented.

[0114] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0118] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for address standardization, characterized by: The method comprises the following steps: Receive the address business table pushed by each business system, extract the relevant data in the address business table into the personnel-related address intermediate table, and insert the ID card information associated with each address business table into the address ID card processing status table of the database; Establish an index of confirmed standard addresses and an index of pending standard addresses in the search engine; at the same time, initialize a set of standard address information, and pull all collected standard address information that meets the standard specifications into the confirmed standard address index as the initial address; Based on the address ID card processing status table, the personnel-related address intermediate table, the determined standard address index, and the pending standard address index, all address information in the personnel-related address intermediate table is standardized using personnel aggregation. This includes calling the personnel address standardization interface using the ID number. The specific operation method of the personnel address standardization interface is addressHandle(zjhm), where zjhm specifically refers to the ID number. The specific operation steps are as follows: Step B1: Initialize a JSON object obj and set the initial return value as a success flag. At the same time, set a failure flag. If an exception occurs during the execution process, the failure flag is returned. Step B2: Initialize an empty string set dzSet, which is used to store addresses that can be used as standard addresses; Step B3: query the personnel-associated address intermediate table based on the ID number, and obtain all address arrays associated with the ID number from the personnel-associated address intermediate table, thereby obtaining a JSON array dzList; Step B4: Process the JSON array dzList as follows: Step B41, obtain the i-th associated address object gldz in the JSON array dzList; Step B42: Get the address details field of the associated address object gldz and assign it to the current address details str; Step B43: Initialize a JSON object maxBzObj and a variable maxBzScore=0, wherein the variable maxBzScore is used to store the maximum similarity, and the JSON object maxBzObj is used to store the standard address information of the maximum similarity; Step B44: Loop through other addresses in the JSON array dzList and compare them with the current address details str to find a suitable standard address object maxBzObj; Step B45: If maxBzScore is less than or equal to 0.9 or the standard address confidence bzscore is -1, the current address details str are standardized according to the region mapping table of the personnel address standardization interface to obtain a temporary standard address. The standard address corresponding to the associated address object gldz is set as the temporary standard address, the standard address confidence bzscore is set to 1, and the temporary standard address is stored in the string set dzSet. If maxBzScore is greater than 0.9, the standard address and maxBzScore in the standard address object maxBzObj are written to the standard address and standard address confidence bzscore corresponding to the associated address object gldz, so as to facilitate the subsequent update of the personnel-associated address intermediate table; at the same time, the standard address with a maxBzScore greater than 0.9 is stored in the string set dzSet; Step B46: Return to step B41 and continue to obtain the next associated address object gldz for processing until the standard address and standard address confidence bzscore are searched for each associated address object gldz in the JSON array dzList. At this point, each associated address object gldz in the JSON array dzList has two additional attributes: standard address and standard address confidence. Step B47: Update the personnel-related address intermediate table according to the attributes of each related address object gldz in the JSON array dzList; Step B48: Loop to determine whether the standard address string in the string set dzSet is in the determined standard address index. If so, no processing is performed. If not, the standard address string is inserted into the standard address index to be reviewed.

2. The address standardization method according to claim 1, wherein: The specific method of extracting relevant data from the address business table to the personnel-associated address intermediate table is as follows: using NiFi to extract relevant data from the address business table to the personnel-associated address intermediate table, the personnel-associated address intermediate table is stored in a search engine, and the personnel-associated address intermediate table includes the following fields: information primary key number, ID number, name, contact number, address code, address details, administrative division where the address is located, address type, longitude, latitude, registration time, number of associations, storage time, standardized address or standard address confidence; wherein, the information primary key number is obtained by performing MD5 operation on the ID number and address details; the value of the address type: source table.

3. The address standardization method according to claim 1, wherein: The address ID card processing status table includes the following fields: ID number, whether processed, storage time, processing time or priority.

4. The address standardization method according to claim 1, wherein: The determined standard address index and the pending standard address index both include the following fields: information primary key number, standard address, standard address ngram segmentation, standard address ik segmentation, address type, longitude, latitude, province, city, county, township or village, wherein the information primary key number is obtained by performing MD5 operation on the address details.

5. The address standardization method according to claim 1, wherein: The standardization of all address information in the personnel-associated address intermediate table by personnel aggregation based on the address ID card processing status table, the personnel-associated address intermediate table, the determined standard address index, and the pending standard address index specifically includes: (1) Use QueryDatabaseTable to pull the data to be processed from the address ID card processing status table, convert the Avro format file pulled by QueryDatabaseTable into JSON array format through the ConvertAvroToJSON component, and then use the SplitJson component to convert the data to be processed into independent JSON data files, and use EvaluateJsonPath to extract the ID card attribute in the JSON data file, that is, the ID number; (2) The personnel address standardization interface is an HTTP interface, the input parameter is the ID number, and the operation result is returned through the personnel address standardization interface, and the operation result is a success mark or a failure mark; (3) Get the operation result through EvaluateJsonPath and update the operation result to the document number corresponding to the address ID card processing status table.

6. The address standardization method according to claim 5, characterized in that: The specific processing flow of the personnel address standardization interface also includes the following: A. Store the complete data of districts, counties, towns, villages and their abbreviations of the entire prefecture-level city into the JDK memory in a map format to obtain an area mapping table. The area mapping table is used as a global parameter of the personnel address standardization interface. The naming format of the area mapping table is prefecture-level city abbreviation + areaMap; Step B44 specifically includes: Step B441, obtain the address details qtdzstr of other current addresses; Step B442: Call the similarAddr(str,qtdzstr,dzSet) method to obtain the compared standard address information. The return object bzObj of the similarAddr(str,qtdzstr,dzSet) method includes the standard address and the standard address confidence score bzscore. Step B443: If the standard address confidence score bzscore is -1, it means that the current address detail str is not a detailed address, and the other addresses in the JSON array dzList are not traversed. If the standard address confidence bzscore is greater than maxBzScore, the standard address object maxBzObj is updated to the return object bzObj of the similarAddr(str,qtdzstr,dzSet) method, and maxBzScore is set to bzscore; if the standard address confidence bzscore is equal to maxBzScore, the length of the standard address object maxBzObj is compared with that of the standard address in the return object bzObj, and the standard address with the longer length is taken as the standard address of the standard address object maxBzObj. If the lengths are the same, the standard address object maxBzObj is not changed; at the same time, return to step B441 to continue to obtain the address details qtdzstr of the next other address for comparison until all other addresses in the JSON array dzList are traversed.

7. The address standardization method according to claim 6, characterized in that: The calling of the similarAddr(str,qtdzstr,dzSet) method to obtain the compared standard address information specifically includes: Step C1: Initialize a JSON object res, which is used to store return parameters, set the initial value of the standard address to the input address string str, and set the standard address confidence bzscore to 0; Step C2: Use the HanLP word segmenter to perform word segmentation on the input address string str and another address string qtdzstr, thereby obtaining a word segmentation array; Step C3: Based on the word segmentation results, the address string str and another address string qtdzstr are standardized using the region mapping table of the personnel address standardization interface; Step C4: Determine whether the standardized address string str is in the string set dzSet. If so, set the standard address confidence bzscore to 1 and directly return the result. If not, determine whether the standardized address string str contains only basic information. If so, set the standard address confidence bzscore to -1 and directly return the result; if not, proceed to step C5; Step C5: Continue to perform word segmentation processing on the standardized address string str and another address string qtdzstr, and determine whether there are numerals in the two address strings. If there are numerals, compare whether the numerals in the two address strings are the same. If the numerals are different, set the standard address confidence score bzscore to 0 and directly return the result; If the numerals are the same, determine whether the two address strings contain each other. If so, select the longer address string as the standard address, set the standard address confidence bzscore to 1, and directly return the result. If the numerals do not exist or the two address strings do not contain each other, proceed to step C6. Step C6: Use the Jaccard similarity algorithm to calculate the similarity value of the two address strings, use the longer address string as the standard address, set the standard address confidence bzscore as the similarity value, and return the result.

8. The address standardization method according to claim 6, characterized in that: The method also includes: obtaining the standard address to be reviewed from the standard address to be reviewed index, and providing several alternative addresses with the closest similarity to the standard address to be reviewed through the word segmentation field in the determined standard address index, and if it is determined after review that the standard address to be reviewed is the same as one of the alternative addresses, then replacing the corresponding standard address in the personnel-associated address intermediate table with the alternative address; if it is determined after review that the standard address to be reviewed is different from both the given alternative addresses, then inserting the standard address to be reviewed into the determined standard address index as the standard address.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images