Address Library Construction Method and Device

Abstract

The present disclosure provides an address library construction method and apparatus. The method includes: obtaining original address data; performing structured processing on the original address data to obtain a structured address data set; performing noise filtering on the structured address data set to obtain a denoised structured address data set; and performing standardization processing on the denoised structured address data set to obtain a standardized address data set. The address library construction method of the present disclosure does not rely on manual editing and annotation, nor on externally existing standard address data. Instead, it identifies noise information in the address data, filters incorrect element relationships, and resolves element relationship conflicts through the characteristics of the original address data itself.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a method and apparatus for constructing an address library. Background Art

[0002] Many methods for constructing an address library rely on a large amount of manual intervention, including manual collection, editing, cleaning, and annotation. That is, the collection of addresses required in the address library is operated by humans, and the inclusion and inclusion relationships between different addresses are determined manually. Moreover, the removal of incorrect addresses in the collected addresses is also manually operated. For example, incorrect matching relationships such as Suzhou City in Henan Province are judged by humans. The annotation of addresses is to match the geographical locations corresponding to the addresses to establish a correspondence, and the existing address libraries use manual matching operations on maps during annotation. Many methods for constructing an address library rely on an existing standard address library (or standard address information data in the form of homogeneous external knowledge, or a preset administrative division table, etc.), and can only process non-standard address data by matching non-standard addresses (i.e., addresses containing noise information) with standard address information. Some methods do not provide clear solutions to common but important problems in actual scenarios such as the elements judged as noise in the address data set, address information conflicts, and address error correction. Summary of the Invention

[0003] The present disclosure provides a method and apparatus for constructing an address library, which are used to solve the defects in the prior art that rely on manual standards, require a standard address library to be established in advance, and do not process noise, etc., and realize that it does not rely on manual editing and annotation, nor on existing external standard address data, but through the characteristics of the original address data itself, identify noise information in the address data, filter incorrect element relationships, and solve element relationship conflicts.

[0004] In a first aspect, the present disclosure provides a method for constructing an address library, including:

[0005] Obtaining original address data;

[0006] Performing structured processing on the original address data to obtain a structured address data set;

[0007] Performing noise filtering on the structured address data set to obtain a denoised structured address data set;

[0008] Performing standardization processing on the denoised structured address data set to obtain a standardized address data set.

[0009] According to a method for constructing an address library provided by the present disclosure, wherein the performing structured processing on the original address data to obtain a structured address data set specifically includes:

[0010] Process the original address data through a named entity recognition model to obtain corresponding address element phrases; wherein, the address element phrases include at least one set of corresponding element types and element texts.

[0011] Determine one or more of the element texts from the address element phrases as central elements.

[0012] Generate the structured address data set by combining at least one set of corresponding element types and element texts and the central elements.

[0013] According to an address library construction method provided by the present disclosure, wherein the noise filtering of the structured address set to obtain a denoised structured address data set specifically includes:

[0014] Divide the element texts in all structured address sets according to the element types to obtain multiple types of element text sets.

[0015] Aggregate each type of the element text sets to obtain an aggregated element text set.

[0016] Perform noise filtering on each type of the aggregated element text sets to obtain a denoised structured address data set.

[0017] According to an address library construction method provided by the present disclosure, wherein the normalization processing of the denoised structured address data set to obtain a normalized address data set specifically includes:

[0018] Extract element triples from the denoised structured address data set to obtain extracted element triples.

[0019] Combine the element triples with a triple template to obtain element triples that conform to the template relationship.

[0020] Judge the true triple relationship of the element triples that conform to the template relationship to determine whether the true triple relationship of the element triples is correct.

[0021] If the true triple relationship of the element triples is incorrect, perform element replacement and correction on the element triples to obtain normalized address data.

[0022] According to an address library construction method provided by the present disclosure, wherein the aggregation of each type of the element text sets to obtain an aggregated element text set specifically includes:

[0023] Aggregate the element text by using an element aggregation method based on semantics supplemented by spatial range constraints to obtain an aggregated set of element texts.

[0024] According to an address library construction method provided by the present disclosure, wherein, filtering noise from each type of the aggregated set of element texts to obtain a denoised structured address data set specifically includes:

[0025] Obtain the frequency of each type of element text in the set of element texts;

[0026] Compare the frequency with a preset threshold to determine the magnitude relationship between the frequency and the threshold;

[0027] If the frequency is smaller than the threshold, determine that the element text is noise;

[0028] Delete the element text determined to be noise to obtain a denoised structured address data set.

[0029] According to an address library construction method provided by the present disclosure, wherein, judging the true triple relationship of the element triple that conforms to the template relationship to determine whether the true triple relationship of the element triple is correct specifically includes:

[0030] Record the instance of the element triple relationship and count the frequency of the element triple that conforms to the template relationship to obtain the frequency of the element triple relationship instance;

[0031] Use the method of truth discovery for the frequency to judge whether the true triple relationship of the element triple is correct.

[0032] According to an address library construction method provided by the present disclosure, wherein, if the true triple relationship of the element triple is incorrect, perform element replacement and correction on the element triple to obtain standardized address data, specifically including:

[0033] Obtain the element triple including the central element of the address data;

[0034] Starting from the element triple including the central element, arrange all the element triples of the address data from coarser to finer in terms of granularity and judge whether the head element in the element triple is correct;

[0035] If the head element is incorrect, replace the head element in the element triple with the tail element in the element triple with a coarser granularity than the element triple and continue to judge whether the head element of the element triple with a coarser granularity is correct;

[0036] If the head element is correct, the element triple including the central element is not replaced, and it continues to determine whether the head element of the element triple with a coarse granularity is correct.

[0037] According to an address library construction method provided by the present disclosure, wherein, clearing the element text determined to be noise to obtain a denoised structured address data set specifically includes:

[0038] If the element determined to be noise is the central element, the entire address data is discarded;

[0039] If the element determined to be noise is not the central element, only the element text is discarded.

[0040] According to an address library construction method provided by the present disclosure, wherein, using the method of truth discovery for the frequency to determine whether the true triple relationship of the element triple is correct, specifically includes:

[0041] Preliminarily filtering the element triple to obtain a preliminarily filtered element triple;

[0042] Determining the uniqueness of the element relationship of the filtered element triple;

[0043] For the element triple with a one-to-one relationship of uniqueness, using the method of frequency statistics to determine that the element triple with the most frequent one-to-one relationship is the element triple with a correct relationship, and then determining that the element triples with other frequencies of the one-to-one relationship are element triples with incorrect relationships;

[0044] For the element triple with a many-to-one relationship of uniqueness, using the element triple in which the confidence of the head element is higher than a predetermined parameter as the element triple with a correct relationship, and then determining that the element triple with a confidence not higher than the predetermined parameter is the element triple with an incorrect relationship.

[0045] In a second aspect, the present disclosure provides an address library construction device, including:

[0046] A first processing module, configured to obtain original address data;

[0047] A second processing module, configured to perform structured processing on the original address data to obtain a structured address data set;

[0048] A third processing module, configured to perform noise filtering on the structured address data set to obtain a denoised structured address data set;

[0049] A fourth processing module, configured to perform standardization processing on the denoised structured address data set to obtain a standardized address data set.

[0050] In a third aspect, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of any one of the above-mentioned address library construction methods are implemented.

[0051] In a fourth aspect, the present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned address library construction methods are implemented.

[0052] An address library construction method and apparatus provided by the present disclosure obtain original address data; then, perform structured processing on the original address data to obtain a structured address data set; after performing structured processing on the data, noise filtering can be performed on the structured address data set according to the structure and type of the data to obtain a denoised structured address data set; thereby reducing the influence of address description diversification and long-tail data problems on the determination of elements determined as noise. Finally, perform standardization processing on the denoised structured address data set to obtain a standardized address data set. The present disclosure realizes the construction of a data set by identifying noise information and standardization processing in address data through the characteristics of the original address data itself, without relying on manual editing and annotation, nor relying on externally existing standard address data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 is one of the flowcharts of the address library construction method provided by the present disclosure;

[0055] Figure 2 is the second flowchart of the address library construction method provided by the present disclosure;

[0056] Figure 3 is the flowchart of performing noise filtering on the structured address set provided by the present disclosure to obtain a denoised structured address data set;

[0057] Figure 4 is the flowchart of obtaining a standardized address data set provided by the present disclosure;

[0058] Figure 5It is a schematic flowchart for noise filtering of the aggregated element text sets of each type provided by the present disclosure to obtain a denoised structured address data set;

[0059] Figure 6 It is a schematic structural diagram of an address library construction device provided by the present disclosure;

[0060] Figure 7 It is a schematic structural diagram of an electronic device provided by the present disclosure. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are some but not all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without making creative efforts belong to the scope protected by the embodiments of the present disclosure.

[0062] The following combines Figure 1 - Figure 2 to describe the address library construction method according to the embodiments of the present disclosure, including:

[0063] Step 100: Obtain original address data;

[0064] Specifically, the objective of the present disclosure is to construct an address library, and the richness of the original address data set is an important factor determining the quality and coverage of the final address library. Therefore, rich original address data should be collected as much as possible to ensure the data basis.

[0065] Step 200: Perform structured processing on the original address data to obtain a structured address data set;

[0066] Specifically, in the present disclosure, since it is necessary to perform noise removal and other processing on the collected address data, and the processing of address data in the present disclosure is based on structured data. Therefore, in the present disclosure, the original address data is structured.

[0067] Step 300: Perform noise filtering on the structured address data set to obtain a denoised structured address data set;

[0068] Specifically, noise introduced by certain factors in the process of generating address data (such as deliberately forged addresses by e-commerce fraud, incorrect or missing user entries, inaccurate user geographical cognition, etc.), or errors and omissions introduced in previous steps such as address structuring, the obtained address element set often contains a considerable number of elements judged as noise or descriptions of long-tail niche elements. Therefore, it is necessary to filter the elements judged as noise. Through the filtering process, a denoised structured address data set is obtained.

[0069] Step 400: Standardize the denoised structured address dataset to obtain a standardized address dataset.

[0070] Specifically, after denoising the structured dataset, there are still errors in the relationships between the data in the dataset. Therefore, it is necessary to standardize the data to obtain a standardized dataset.

[0071] An address library construction method provided by the present disclosure includes obtaining original address data; then, performing structured processing on the original address data to obtain a structured address dataset; after performing structured processing on the data, noise filtering can be performed on the structured address dataset according to the structure and type of the data to obtain a denoised structured address dataset; thereby reducing the influence of address description diversification and long-tail data problems on the determination of elements determined as noise. Finally, standardize the denoised structured address dataset to obtain a standardized address dataset. The present disclosure realizes the construction of the dataset by identifying noise information and standardization processing in the address data through the characteristics of the original address data itself, without relying on manual editing and annotation, nor relying on existing standard address data externally.

[0072] According to an address library construction method provided by an embodiment of the present disclosure, wherein the performing structured processing on the original address data to obtain a structured address dataset specifically includes:

[0073] Processing the original address data through a named entity recognition model to obtain corresponding address element phrases; wherein the address element phrases include at least one set of corresponding element types and element texts;

[0074] Determine one or more of the element texts from the address element phrases as the central element;

[0075] Generate the structured address dataset according to at least one set of corresponding element types and element texts and the central element.

[0076] Specifically, for each address, by using a named entity recognition model (Named Entity Recognition) for address entities to process the address text, address element phrases with geographical significance are obtained, which can be described symbolically as: f(Addr) = [(element type 1: element text 1), (element type 2: element text 2),..., (element type n: element text n)].

[0077] Among them, Addr represents the original address, and the function f represents the structured processing operation of the named entity recognition model on the address data. The right side of the equation is the structured result (a list of address elements of different types). For example, for the original address text "Room C, Unit B, Building A, West Area of Nanmouyuan New Village, Yangmoushe Town, Zhangjiagang City, Jiangsu Province", the structured result is multiple geographical elements of different types: "Province: Jiangsu Province|City: Zhangjiagang City|County: Zhangjiagang City|Town: Yangmoushe Town|Residence: West Area of Nanmouyuan New Village|BuildNum: Building A|Unit: Unit B|Room: Room C".

[0078] Regarding the named entity recognition model, reasonable geographical element types can be defined according to specific data scenarios. The following shows a classification method for address element types applicable to express logistics address data, and the markings and meanings of some address elements are shown in Table 1 below:

[0079] Table 1

[0080]

[0081] Furthermore, the central element (POI) is selected from multiple elements of each address.

[0082] In this article, the element (or combination of elements) representing the address destination or specific point of interest is defined as the central element of the address, which represents the geographical element that people are most concerned about in the entire address text, that is, the POI (Point of Interest). For example: the central element of the address "Province: Jiangsu Province|City: Zhangjiagang City|County: Zhangjiagang City|Town: Yangmoushe Town|Residence: West Area of Nanmouyuan New Village|BuildNum: Building A|Unit: Unit B|Room: Room C" is "Residence: West Area of Nanmouyuan New Village"; the central element of the address "City: Beijing|County: Daxing District|Road: A Certain Shiyi Street|Institution: A Certain Building|BuildNum: Building A" is "Institution: A Certain Building".

[0083] Combined with Figure 3 shown, and referring to Figure 2 According to an address library construction method provided by an embodiment of the present disclosure, wherein, the noise filtering of the structured address set to obtain a denoised structured address data set specifically includes:

[0084] Dividing the element texts in all structured address sets according to element types to obtain multiple types of element text sets;

[0085] Aggregate the set of the element texts of each type to obtain an aggregated set of element texts;

[0086] Perform noise filtering on the aggregated set of element texts of each type to obtain a denoised structured address data set.

[0087] Specifically, divide the set of each type of element text. For all address elements, divide the elements according to the element type to obtain a set of elements of each type. The form of the set of elements is as follows:

[0088] Element set = {Province: [Guangmou Province, Guangmou, Jiangmou, Xinmou, Xinmou Autonomous Region, Beimou,..., Shanmou Province],

[0089] City: [Sumi City, Sumu, Sumi City, Hangmou, Nanmou, Shenmou,..., Beimou],

[0090] County: [Daxing District, Tianhe District, Longgang District, Xuhui District,..., Xicheng District],

[0091] Institution: [F Headquarters, F Building, G County Snacks,..., F Police Station],

[0092] ...};

[0093] Aggregating the above element sets means placing elements that use different representations of elements but can be placed in the same element set into the same subset. The aggregated result is as follows:

[0094] Institution: ([Beijingmou Headquarters, Beijingmou Building, Beijingmou Mansion, Beijingmou Company Headquarters], [Tongmou Second Hospital, Tongmou Second Hospital],...);

[0095] Residence: ([Fumou Shangyueju, Fumou Community, Fumou Shangyueju Community], [Xingmou International, Xingmou International Community, Xingmou Community],...)

[0096] Among them, the reasons for noise generation include noise introduced by certain factors in the process of generating address data (such as deliberately forging addresses by e-commerce brush orders, incorrect or missing user filling, inaccurate user geographical cognition, etc.), or errors and omissions introduced in the previous steps such as address structuring. As a result, the obtained set of address elements often contains a significant number of elements judged as noise or descriptions of long-tail niche elements.

[0097] For each type of element, a noise data filtering module for specific types of elements is constructed, and then the noise filtering module is used to determine whether each aggregated subset is a noise subset one by one, and determine whether each subset is retained. For example, for the element subset "[Jingmou Headquarters, Jingmou Building, Jingmou Mansion, …, Jingmou Company Headquarters]", if it is determined that this subset is a noise subset, then all elements in this subset are discarded; if it is determined that it is not a noise subset, then all elements of the subset are retained.

[0098] For the retained element subsets, a unified ID is assigned to all elements within the subset. The unified ID indicates that the elements within a subset have a certain spatial proximity and a high semantic similarity. For example, the same ID is assigned to each element within the subset "[Jingmou Headquarters, Jingmou Building, Jingmou Mansion, Jingmou Company Headquarters]".

[0099] Combined Figure 4 as shown, and referring to Figure 2 , according to a method for constructing an address library provided by an embodiment of the present disclosure, wherein, the denoised structured address data set is standardized to obtain a standardized address data set, specifically including:

[0100] Extract element triples from the denoised structured address data set to obtain the extracted element triples;

[0101] Combine the element triples with the triple template to obtain element triples that conform to the template relationship;

[0102] Judge the true triple relationship of the element triples that conform to the template relationship, and judge whether the true triple relationship of the element triples is correct;

[0103] If the true triple relationship of the element triples is incorrect, then perform element replacement and correction on the element triples to obtain standardized address data.

[0104] Specifically, after denoising the structured data set, the relationship between elements is described by element triples. An element triple refers to a data structure instance in the form of <element h, relationship r, element t>, where in the following, element h is called the head element and element t is called the tail element. For example: <Jiangmou Province, contains, Sumou City>, <Sumou City, contains, Hu District>, <Gu District, intersects, Ren Road>, <Xiang District, intersects, Ren Road>, <Ke 11th Street, intersects, Jingmou Headquarters>.

[0105] Due to people's insufficient understanding of geographical information, inadvertent filling errors, deliberate forgery, or other reasons during the generation of address data, the original address data often contains incorrect element relationships, such as <Zhejiang Province, contains, Guangzhou City>. For the entire address dataset, it presents element relationships with description conflicts. For example, among all the addresses containing Nantong City, most of the addresses filled in by users can extract the element triple <Jiangsu Province, contains, Nantong City>, but at the same time, there is a small proportion of addresses filled in by users with the element relationship <Zhejiang Province, contains, Nantong City>. For the element triples with description conflicts, alignment is required for error correction or filtering.

[0106] Several common element relationships mainly include three types: contains, intersects, and belongs to.

[0107] The contains relationship mainly means that element h completely contains element t in terms of spatial scope. For example, a provincial administrative region contains multiple municipal administrative regions, a municipal administrative region contains multiple districts and counties, and an administrative region contains multiple short roads;

[0108] The intersects relationship refers to the relationship where two elements intersect in space but do not contain each other. In particular, in this article, when two elements are tangent in space, it is also called intersecting. Specifically, for example, if two administrative regions A1 and A2 are divided by road B, then it is said that A1 and B intersect, and A2 and B also intersect; another example is that Komuchuang 11th Street, Yard D is the road coding description of Jing's headquarters. In this case, this article also says that Komuchuang 11th Street and Jing's headquarters intersect. The intersects relationship is common in the case where Road type elements intersect with various coarse-grained administrative regions, and is also common in the case where Road type elements intersect with various fine-grained elements such as Build / School / Residence / Institution. For example, for the address "City: Beijing | County: Daxing District | Road: Komu 11th Street | Institution: Jing's Building | BuildNum: Block A", the element triple "<Komu 11th Street, intersects, Jing's Building>" can be extracted; for the address "City: Beijing | County: Tongzhou District | Town: Majuqiao Town | Road: Xingmao 3rd Street | Institution: Xingyue International Community | BuildNum: Block A", the element triple "<Xingmao 3rd Street, intersects, Xingyue International Community>" can be extracted.

[0109] The belongs to relationship mainly refers to the relationship between a road and a secondary road / service road, as well as other element relationships that conform to the popular meaning of belonging and do not fall within the scope of the aforementioned contains / intersects relationships.

[0110] Among them, an element triple template is established. The form of the element triple template is shown in Table 2 below:

[0111] Table 2

[0112]

[0113] Extract the instance of the element triple of all addresses and conduct statistics. Traverse all addresses in the address database. For each address, traverse the pairwise adjacent elements. For the instance of the element relationship that conforms to the element triple template, record and count the frequency. In particular, when conducting the frequency statistics of the triple instances, for the element IDs described above, merge and count the triples whose element texts are not exactly the same but have the same element ID. For example, assume that 4 triples of element texts are extracted from 4 addresses: <Jiangmou Province, contains, Nanmou City>, <Jiangmou, contains, Nanmou City>, <Jiangmou Province, contains, Nanmou>, <Jiangmou, contains, Nanmou>. Although the texts are not exactly the same, since ID(“Jiangmou”) = ID(“Jiangmou Province”) and ID(“Nanmou City”) = ID(“Nanmou”), the above 4 triple instances can be merged into a unified element triple <ID(“Jiangmou”), contains, ID(“Nanmou”)>, and the frequency is accumulated to 4. The statistical result of this strategy is beneficial to judging the correct element relationship and solving the element relationship conflict.

[0114] If it is determined that the true triple relationship of the element triple is incorrect, then perform element replacement and correction on the element triple to obtain standardized address data.

[0115] According to an address database construction method provided by an embodiment of the present disclosure, wherein, the aggregating each type of the element text set to obtain an aggregated element text set specifically includes:

[0116] Aggregate the element texts by using an element aggregation method based on semantics supplemented by spatial range constraints to obtain an aggregated element text set.

[0117] Specifically, aggregate the elements based on semantic similarity and combined with spatial proximity. For each type of element set, perform element aggregation based on semantics and combined with a certain spatial range constraint within the set, so that highly similar elements are assigned to the same subset. The obtained subset results are in the form of:

[0118] Institution: ([Jingmou Headquarters, Jingmou Building, Jingmou Mansion, Jingmou Company Headquarters], [Tongmou Second Hospital, Tongmou Second Hospital], …);

[0119] Residence: ([Fumou Shangyueju, Fumou Community, Fumou Shangyueju Community], [Xingmou International, Xingmou International Community, Xingmou Community], …)

[0120] Among them, for the spatial range constraint conditions, different constraint conditions should be determined according to different types of element granularity and the actual data conditions available. For example, for geographical elements with a very large spatial granularity such as the Province or City type and a small text diversity, spatial constraints can be ignored during aggregation; for geographical elements with a small spatial granularity such as Institution, Residence, School, etc. and a large text diversity, the spatial constraint conditions can be set to aggregate within a local area of a scale of several kilometers or to aggregate within the scope of the next higher-level administrative region during aggregation.

[0121] Combined Figure 5 as shown, and with reference to Figure 2 According to an address library construction method provided by an embodiment of the present disclosure, wherein, the noise filtering of the aggregated element text set of each type to obtain a denoised structured address data set specifically includes:

[0122] Obtain the frequency of each type of element text in the element text set;

[0123] Compare the frequency with a preset threshold to determine the magnitude relationship between the frequency and the threshold;

[0124] If the frequency is smaller than the threshold, determine that the element text is noise;

[0125] Clear the element text determined to be noise to obtain a denoised structured address data set.

[0126] Specifically, the determination method of the noise subset generally takes the number of addresses associated with the element subset and the element type as the main features. The more addresses are associated with the element subset, the higher the authenticity of the element subset and the lower the possibility of belonging to the noise subset. Therefore, a simple implementation strategy is to set a frequency threshold for each type of element, and determine the element with a frequency lower than the threshold as the element determined to be noise. When other data conditions related to the address are available, the characteristics of other data can also be integrated into the determination module of the element determined to be noise. For example, in common scenarios such as logistics addresses and e-commerce delivery addresses, the more users are associated with the address element subset and the more frequent the online transactions are, the higher the authenticity of the address element subset and the lower the possibility of belonging to the noise. For the element subset retained in the previous step, assign a unified ID to all elements within the subset. The unified ID indicates that the elements within a subset have a certain spatial proximity and a high semantic similarity.

[0127] Furthermore, in the present disclosure, the element text determined to be noise is cleared to obtain a denoised structured address data set.

[0128] An address library construction method provided according to an embodiment of the present disclosure, wherein, for the element triples that conform to the template relationship, determining whether the true triple relationship of the element triples is correct specifically includes:

[0129] Recording and statistically analyzing the element triple relationship instances of the element triples that conform to the template relationship to obtain the frequency of the element triple relationship instances;

[0130] Using the method of truth discovery for the frequency to determine whether the true triple relationship of the element triples is correct.

[0131] Specifically, extracting and statistically analyzing the element triple instances of all addresses. Traverse all addresses in the address database. For each address, traverse two adjacent elements. For the element relationship instances that conform to the element triple template, record and statistically analyze them. In particular, when performing the statistical analysis of the triple instance frequency, for the element IDs mentioned above, merge and statistically analyze the triples with inconsistent element texts but the same element IDs. For example, assume that <Province of Jiang, contains, City of Nan>, <Jiang, contains, City of Nan>, <Province of Jiang, contains, Nan>, <Jiang, contains, Nan> are extracted from 4 addresses, and these 4 element triples with inconsistent texts. However, since ID(“Jiang”)=ID(“Province of Jiang”), ID(“City of Nan”)=ID(“Nan”), the above 4 triple instances can be merged into a unified element triple <ID(“Jiang”), contains, ID(“Nan”)>, and the frequency is accumulated to 4. The statistical results of this strategy are beneficial to judging the correct element relationship and resolving element relationship conflicts.

[0132] In the order from coarser to finer element granularity, using methods such as truth discovery to resolve conflicts and filter out noise relationships for the element triples in the previous step to obtain the correct element relationships, that is, the set of correct element triples.

[0133] Among them, processing in the order from coarser to finer element granularity means that when confirming the relationships of multiple address elements with hierarchical dependencies, give priority to processing elements with a larger spatial scope. For example, first confirm the relationship between Province (provincial administrative region) and City (municipal administrative region), then confirm the relationship between City (municipal administrative region) and County (district or county), and then confirm the relationship between County (district or county) and Town (town), …, and so on.

[0134] An address library construction method provided according to an embodiment of the present disclosure, wherein, if the true triple relationship of the element triples is incorrect, then performing element replacement and correction on the element triples to obtain standardized address data, specifically including:

[0135] Obtain an element triple including a central element of the address data;

[0136] Starting from the element triple including the central element, arrange all the element triples of the address data from coarser to finer in terms of granularity, and determine whether the head element in the element triple is correct;

[0137] If the head element is incorrect, replace the head element in the element triple with the tail element in the element triple with a coarser granularity than the element triple, and continue to determine whether the head element in the element triple with the coarser granularity is correct;

[0138] If the head element is correct, do not replace the element triple including the central element, and continue to determine whether the head element in the element triple with the coarser granularity is correct.

[0139] Specifically, since a structured address is an ordered sequence of elements, a list of element triples that depend on each other before and after can be extracted from it. For example, Example A:

[0140] The structured address "Province: Zhejiang|City: Suzhou City|County: Zhangjiagang City|Town: Yangshe Town|Residence: West Area of Nan Yuan New Village" can extract multiple element triples that depend on each other before and after: ① <Jiangsu, contains, Suzhou City>, ② <Suzhou City, contains, Yangshe Town>, ③ <Yangshe Town, contains, West Area of Nan Yuan New Village>, where the central element of this address is West Area of Nan Yuan New Village.

[0141] Since multiple element triples of an address depend on each other before and after, we set the principle of the order of element triples for error correction as: starting from the element triple including the central element, in the order from finer to coarser in terms of granularity, judge and correct each element triple one by one. For example, for the above Example A, start from the ③rd triple <Yangshe Town, contains, West Area of Nan Yuan New Village> including the central element, determine whether West Area of Nan Yuan New Village is contained in Yangshe Town. If not, replace Yangshe Town in the triple and the previous triple (③ and ②) with the correct town-level administrative region; then judge triple ②, and then ①.

[0142] After correcting all addresses, remove duplicates from the corrected address set to obtain the final address database.

[0143] According to an address database construction method provided by an embodiment of the present disclosure, wherein, the clearing of the element text determined to be noise to obtain a denoised structured address data set specifically includes:

[0144] If the element determined to be noise is the central element, discard the entire address data;

[0145] If the element determined to be noise is not the central element, only discard the element text.

[0146] Specifically, if the central element of an address belongs to the element determined to be noise, the entire address should be discarded; if the central element of an address does not belong to the element determined to be noise, only discard the elements belonging to noise in the address, and retain the non-noise elements.

[0147] According to an address database construction method provided by an embodiment of the present disclosure, wherein, the true value discovery method is used for the frequency to determine whether the true triple relationship of the element triple is correct, specifically including:

[0148] Perform preliminary filtering on the element triple to obtain a preliminarily filtered element triple;

[0149] Determine the uniqueness of the element relationship of the filtered element triple;

[0150] For the element triple with a one-to-one relationship in terms of uniqueness, use the frequency statistics method to determine that the element triple with the most frequent one-to-one relationship is the element triple with a correct relationship, and then determine that the element triples with other frequencies in the one-to-one relationship are element triples with incorrect relationships;

[0151] For the element triple with a many-to-one relationship in terms of uniqueness, use the element triple in which the confidence of the head element is higher than a predetermined parameter as the element triple with a correct relationship, and then determine that the element triple with a confidence not higher than the predetermined parameter is the element triple with an incorrect relationship.

[0152] Specifically, the first sub-step is to preliminarily filter noise triples. Set a loose frequency threshold for each element triple template, and determine the element relationship of the element triple with a frequency less than the threshold as the element relationship of noise to be filtered out;

[0153] The second sub-step is to confirm the uniqueness of the element relationship corresponding to each element triple template. Specifically, divide the element relationship into a one-to-one relationship and a many-to-one relationship. For example, a city only belongs to the upper-level provincial administrative region, so the element relationship described by the template <Province, contains, County> is a one-to-one relationship; for the template <Road, intersects, Residence>, a residential area may intersect with multiple roads, so it belongs to a many-to-one relationship.

[0154] The third sub-step is that for the one-to-one element triples, when resolving relationship conflicts, the relationship with the highest occurrence frequency is selected as the correct solution through a voting method. For example, for the problem of <Province X, contains, City Nanmou Tong>, some addresses fill in that "Zhejiang Province" contains "City Nanmou Tong", and some addresses fill in that "Jiangsu Province" contains "City Nanmou Tong". Through voting, it is found that the most users fill in "Jiangsu Province", so it is confirmed that "Province X" should take the value of "Jiangsu Province"; for the many-to-one element triple <element h, relationship r, element t>, assuming that under the condition of fixing the tail element t and the relationship r, the occurrence frequencies of different values of the head element h conform to a normal distribution. Parameters such as the mean and variance of the normal distribution are estimated for each type of element relationship, and then a mathematical hypothesis testing method is adopted to evaluate the confidence levels of different head element values. By setting the significance level parameter α, several high-confidence element relationships can be selected, thus solving the element conflict problem of the many-to-one type.

[0155] Combined with Figure 6 As shown in the figure, the embodiment of the present disclosure provides an address library construction device, including:

[0156] The first processing module 61 is used to obtain the original address data;

[0157] The second processing module 62 is used to perform structured processing on the original address data to obtain a structured address data set;

[0158] The third processing module 63 is used to perform noise filtering on the structured address data set to obtain a denoised structured address data set;

[0159] The fourth processing module 64 is used to perform standardization processing on the denoised structured address data set to obtain a standardized address data set.

[0160] Since the device provided by the embodiment of the present invention can be used to execute the method described in the above embodiment, and its working principle and beneficial effects are similar, they will not be elaborated here. For specific content, please refer to the introduction of the above embodiment.

[0161] An address library construction device provided by an embodiment of the present disclosure obtains original address data; then, performs structured processing on the original address data to obtain a structured address data set; after performing structured processing on the data, it is possible to perform noise filtering on the structured address data set according to the structure and type of the data to obtain a denoised structured address data set; thereby reducing the influence of diversified address descriptions and long-tail data problems on the determination of elements determined as noise. Finally, perform standardization processing on the denoised structured address data set to obtain a standardized address data set. The present disclosure realizes the construction of a data set by identifying noise information and standardization processing in address data through the characteristics of the original address data itself, without relying on manual editing and annotation, nor on externally existing standard address data.

[0162] According to an address library construction device provided by an embodiment of the present disclosure, wherein the second processing module 62 is specifically configured to:

[0163] Process the original address data through a named entity recognition model to obtain corresponding address element phrases; wherein the address element phrases include at least one set of corresponding element types and element texts;

[0164] Determine one or more of the element texts from the address element phrases as central elements;

[0165] Generate the structured address data set according to at least one set of corresponding element types and element texts and the central elements.

[0166] According to an address library construction device provided by an embodiment of the present disclosure, wherein the third processing module 63 is specifically configured to:

[0167] Divide the element texts in all structured address sets according to element types to obtain multiple types of element text sets;

[0168] Aggregate each type of the element text sets to obtain an aggregated element text set;

[0169] Perform noise filtering on each type of the aggregated element text sets to obtain a denoised structured address data set.

[0170] According to an address library construction device provided by an embodiment of the present disclosure, wherein the fourth processing module 64 specifically includes:

[0171] Extract element triples from the denoised structured address data set to obtain the extracted element triples;

[0172] Combine the element triple with the triple template to obtain an element triple that conforms to the template relationship;

[0173] Perform a true triple relationship judgment on the element triple that conforms to the template relationship to determine whether the true triple relationship of the element triple is correct;

[0174] If the true triple relationship of the element triple is incorrect, perform element replacement and correction on the element triple to obtain standardized address data.

[0175] According to an address library construction device provided by an embodiment of the present disclosure, wherein the third processing module 63 is further specifically configured to:

[0176] Aggregate the element text by using an element aggregation method based on semantics supplemented by spatial range constraints to obtain an aggregated set of element texts.

[0177] According to an address library construction device provided by an embodiment of the present disclosure, wherein the third processing module 63 is further specifically configured to:

[0178] Obtain the frequency of each type of element text in the set of element texts;

[0179] Compare the frequency with a preset threshold to determine the magnitude relationship between the frequency and the threshold;

[0180] If the frequency is smaller than the threshold, determine that the element text is noise;

[0181] Delete the element text determined to be noise to obtain a denoised structured address data set.

[0182] According to an address library construction device provided by an embodiment of the present disclosure, wherein the fourth processing module is further specifically configured to:

[0183] Record the element triple relationship instances and perform frequency statistics on the element triple that conforms to the template relationship to obtain the frequency of the element triple relationship instances;

[0184] Use the method of truth discovery for the frequency to determine whether the true triple relationship of the element triple is correct.

[0185] According to an address library construction device provided by an embodiment of the present disclosure, wherein the fourth processing module 64 is further specifically configured to:

[0186] Obtain the element triple including the central element of the address data;

[0187] Starting from the element triple including the central element, arrange all the element triples of the address data from coarser to finer in terms of granularity, and determine whether the head element in the element triple is correct;

[0188] If the head element is incorrect, replace the head element in the element triple with the tail element in the element triple with a coarser granularity than the element triple, and continue to determine whether the head element in the element triple with a coarser granularity is correct;

[0189] If the head element is correct, do not replace the element triple including the central element, and continue to determine whether the head element in the element triple with a coarser granularity is correct.

[0190] According to an address library construction device provided by an embodiment of the present disclosure, wherein the third processing module 63 is further specifically configured to:

[0191] If the element determined to be noise is the central element, discard the entire address data;

[0192] If the element determined to be noise is not the central element, only discard the element text.

[0193] According to an address library construction device provided by an embodiment of the present disclosure, wherein the fourth processing module 64 is further specifically configured to:

[0194] Perform preliminary filtering on the element triple to obtain a preliminarily filtered element triple;

[0195] Determine the uniqueness of the element relationship of the filtered element triple;

[0196] For the element triple with a one-to-one relationship, use the method of frequency statistics to determine that the element triple with the most frequent one-to-one relationship is the element triple with a correct relationship, and then determine that the element triples with other frequencies have incorrect relationships;

[0197] For the element triple with a many-to-one relationship, use the element triple in which the confidence of the head element is higher than a predetermined parameter as the element triple with a correct relationship, and then determine that the element triples with a confidence not higher than the predetermined parameter have incorrect relationships.

[0198] Figure 7 Illustrated is a schematic diagram of the physical structure of an electronic device, such as Figure 7As shown in the figure, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communications interface 720, and the memory 730 complete their mutual communication through the communication bus 740. The processor 710 may call the logical instructions in the memory 730 to execute an address library construction method, which includes: obtaining original address data; performing structured processing on the original address data to obtain a structured address data set; performing noise filtering on the structured address data set to obtain a denoised structured address data set; performing normalization processing on the denoised structured address data set to obtain a normalized address data set.

[0199] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present disclosure, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0200] On the other hand, the present disclosure also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute an address library construction method provided by the above-mentioned various methods. The method includes: obtaining original address data; performing structured processing on the original address data to obtain a structured address data set; performing noise filtering on the structured address data set to obtain a denoised structured address data set; performing normalization processing on the denoised structured address data set to obtain a normalized address data set.

[0201] In another aspect, the present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements an address library construction method provided above, and the method includes: obtaining original address data; performing structured processing on the original address data to obtain a structured address data set; performing noise filtering on the structured address data set to obtain a denoised structured address data set; performing normalization processing on the denoised structured address data set to obtain a normalized address data set.

[0202] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0203] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, and are not intended to limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for constructing an address library, characterized in that Including: Obtain the original address data; Perform structured processing on the original address data to obtain a structured address data set; Perform noise filtering on the structured address data set to obtain a denoised structured address data set; Perform standardization processing on the denoised structured address data set to obtain a standardized address data set; The performing standardization processing on the denoised structured address data set to obtain a standardized address data set specifically includes: Extract element triples from the denoised structured address data set to obtain the extracted element triples; Combine the element triples with a triple template to obtain element triples that conform to the template relationship; Judge the true triple relationship of the element triples that conform to the template relationship to determine whether the true triple relationship of the element triples is correct; wherein, the relationship of the element triples is obtained according to the instance records and frequency statistics of the triple relationship; If the true triple relationship of the element triples is incorrect, perform element replacement and correction on the element triples to obtain standardized address data; Performing noise filtering on the structured address data set to obtain a denoised structured address data set includes: Process the original address data through a named entity recognition model to obtain corresponding address element phrases; wherein, the address element phrases include at least one set of corresponding element types and element texts; Divide the element texts in all structured address sets according to the element type to obtain multiple types of element text sets; Aggregate each type of the element text sets to obtain an aggregated element text set; Perform noise filtering on each type of the aggregated element text sets to obtain a denoised structured address data set.

2. The method for constructing an address library according to claim 1, wherein The performing structured processing on the original address data to obtain a structured address data set specifically includes: Determine one or more of the element texts in the address element phrases as the central elements; Generate the structured address data set according to at least one set of corresponding element types and element texts and the central elements.

3. The method for constructing an address library according to claim 1, wherein The aggregating each type of the element text sets to obtain an aggregated element text set specifically includes: Aggregate the element texts by using an element aggregation method based on semantics and supplemented by spatial range constraints to obtain an aggregated element text set.

4. The method for constructing an address library according to claim 1, wherein The performing noise filtering on each type of the aggregated element text sets to obtain a denoised structured address data set specifically includes: Obtain the frequency of each type of element text in the element text set; Compare the frequency with a preset threshold to judge the magnitude between the frequency and the threshold; If the frequency is smaller than the threshold, judge the element text as noise; Delete the element texts judged as noise to obtain a denoised structured address data set.

5. The method for constructing an address library according to claim 1, wherein The judging the true triple relationship of the element triples that conform to the template relationship to determine whether the true triple relationship of the element triples is correct specifically includes: Record the instance of the element triple relationship and perform frequency statistics on the element triple that conforms to the template relationship to obtain the frequency of the element triple relationship instance. Use the method of truth discovery for the frequency to judge whether the true triple relationship of the element triple is correct.

6. The method for constructing an address library according to claim 1, wherein If the true triple relationship of the element triple is incorrect, perform element replacement and correction on the element triple to obtain standardized address data, specifically including: Obtain the element triple including the central element of the address data. Starting from the element triple including the central element, arrange all the element triples of the address data from coarser to finer in terms of granularity, and judge whether the head element in the element triple is correct. If the head element is incorrect, replace the head element in the element triple with the tail element in the element triple with a coarser granularity than the element triple, and continue to judge whether the head element of the element triple with a coarser granularity is correct. If the head element is correct, do not replace the element triple including the central element, and continue to judge whether the head element of the element triple with a coarser granularity is correct.

7. The method for constructing an address library according to claim 4, wherein Clear the element text judged as noise to obtain a denoised structured address data set, specifically including: If the element judged as noise is the central element, discard the entire address data. If the element judged as noise is not the central element, only discard the element text.

8. The method for constructing an address library according to claim 5, wherein Using the method of truth discovery for the frequency to judge whether the true triple relationship of the element triple is correct, specifically including: Perform preliminary filtering on the element triple to obtain a preliminarily filtered element triple. Determine the uniqueness of the element relationship of the filtered element triple. For the element triple with a one-to-one relationship of uniqueness, use the method of frequency statistics to determine that the element triple with the most frequent one-to-one relationship is the element triple with a correct relationship, and then determine that the element triples with other frequencies of the one-to-one relationship are element triples with incorrect relationships. For the element triple with a many-to-one relationship of uniqueness, use the element triple with the confidence of the head element higher than the predetermined parameter as the element triple with a correct relationship, and then determine that the element triple with the confidence not higher than the predetermined parameter is the element triple with an incorrect relationship.

9. An address library construction device, characterized in that, Including: The first processing module is used to obtain the original address data. The second processing module is used to perform structured processing on the original address data to obtain a structured address data set. The third processing module is used to perform noise filtering on the structured address data set to obtain a denoised structured address data set. The fourth processing module is used to perform standardization processing on the denoised structured address data set to obtain a standardized address data set. The fourth processing module is specifically used to perform element triple extraction on the denoised structured address data set to obtain the extracted element triple. Combine the element triple with the triple template to obtain an element triple that conforms to the template relationship. Perform a true triple relationship judgment on the element triple that conforms to the template relationship, and determine whether the true triple relationship of the element triple is correct; wherein, the relationship of the element triple is obtained according to the instance record and frequency statistics of the triple relationship; If the true triple relationship of the element triple is incorrect, perform element replacement and correction on the element triple to obtain standardized address data; The third processing module is specifically used for: Process the original address data through a named entity recognition model to obtain corresponding address element phrases; wherein, the address element phrases include at least one set of corresponding element types and element texts; Divide the element texts in all structured address sets according to the element types to obtain multiple types of element text sets; Aggregate each type of the element text sets to obtain an aggregated element text set; Perform noise filtering on each type of the aggregated element text sets to obtain a denoised structured address data set.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the address library construction method according to any one of claims 1 to 8 are implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the address library construction method according to any one of claims 1 to 8 are implemented.

Images