User-defined address processing method and device based on address structured analysis

Abstract

The invention discloses a custom address processing method and device based on address structured analysis. The method comprises the following steps: firstly, obtaining an original address input by a user; searching the original address through a distributed address index vector library to obtain a matching address having the highest similarity confidence rate with the original address; complementing the matched address to obtain a target address; carrying out structured analysis on the target address to obtain analysis data; and finally, according to the analysis data, performing self-defined address grading on the service. The method can support automatic custom level labeling according to business requirements or data specifications, thereby providing high flexibility and customizability. Besides, automatic processing is realized in the whole analysis and labeling process, manual intervention is eliminated, efficiency and consistency are improved, high consistency of address labeling results under different users or scenes can be ensured, and the method has wide application value.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a method and device for processing a custom address based on address structured analysis. Background Art

[0002] Address structured parsing technology has been widely used in many fields such as e-commerce, logistics, urban planning and public services.

[0003] However, existing technologies mainly rely on manual work to complete automatic address completion and customized address grading, lacking effective automation methods. This reliance not only has a negative impact on the capabilities and quality of address structuring technology, but also limits its potential for efficiency improvement. Therefore, there is an urgent need to develop more advanced automation solutions to improve the accuracy and efficiency of address processing, so as to better meet various business needs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and propose a custom address processing method and device based on address structured parsing, which can automatically realize address completion and custom address grading, thereby improving the ability, quality and efficiency of address structuring.

[0005] In a first aspect, the present invention provides a method for processing a custom address based on structured address parsing, the method comprising the following steps:

[0006] Step S1: Obtain the original address input by the user;

[0007] Step S2: searching the original address through a distributed address index vector library to obtain a matching address with the highest similarity confidence rate with the original address;

[0008] Step S3: completing the matching address to obtain a target address;

[0009] Step S4: performing structural analysis on the target address to obtain analysis data;

[0010] Step S5: According to the parsed data, the services are classified into custom addresses to complete custom address processing based on address structured parsing.

[0011] Further, before the step S2, the method further includes a step S0;

[0012] Step S0: construct a distributed address index vector library;

[0013] The step S0 specifically includes the following steps:

[0014] Get OSM address data from the OSM open source address information database;

[0015] Preprocessing the OSM address data to make it conform to the index structure of Elasticsearch;

[0016] The preprocessed OSM address data is imported into the Elasticsearch index in batches using the Elasticsearch Bulk API, thereby building a distributed address index vector library with multi-mode search capabilities.

[0017] Furthermore, the step S2 specifically includes the following steps:

[0018] Based on the Tongyi Qianwen 72B large language model, extracting a feature vector corresponding to the original address;

[0019] Use the Elasticsearch index and import the original address data and extracted feature vectors into the Elasticsearch index in batches according to the Elasticsearch Bulk API.

[0020] Based on the k-NN query function of Elasticsearch, a vector search corresponding to the feature vector and the original address is performed by specifying a query vector and a search parameter;

[0021] Receive the address data similar to the original address returned by ElasticSearch, and take the address corresponding to the highest similarity confidence rate as the matching address.

[0022] Furthermore, the step S3 is specifically:

[0023] The matching address is fully completed according to the WGS84 longitude and latitude code to obtain the target address.

[0024] Furthermore, the step S4 is specifically: using the Byzantine Generals Consensus Algorithm to perform structured analysis on the target address to obtain analysis data;

[0025] The Byzantine Generals Consensus Algorithm is used to perform structured analysis on the target address to obtain analysis data, which specifically includes the following steps:

[0026] Parsing the target address through a parsing node to obtain a parsing result;

[0027] Wherein, the parsing node is an independent unit or component for parsing the target address; the parsing result is generated by each parsing node receiving the same target address and independently using the Tongyi Qianwen 72B large language model for parsing;

[0028] Publishing the parsing result in a first network composed of the parsing nodes;

[0029] The Byzantine Generals Consensus Algorithm is adopted to collect information of parsing nodes through the first network for voting, and the parsing result with the highest score is selected as the final parsing result, thereby completing the generation of parsing data.

[0030] Furthermore, in step S5, the business is classified by custom address, specifically: the business is classified by custom address using the Byzantine Generals Consensus Algorithm;

[0031] The Byzantine Generals Consensus Algorithm is used to perform custom address classification on the business, which specifically includes the following steps:

[0032] Classify the business through the classification nodes to obtain the classification results;

[0033] The grading node is an independent unit or component for grading services; the grading result is obtained by each grading node according to the grading standard and independently using the Tongyi Qianwen 72B large language model to determine the level to which the address in the parsed data belongs; the grading standard is pre-set according to business requirements;

[0034] Publishing the grading result in the second network composed of the grading nodes;

[0035] By adopting the Byzantine Generals Consensus Algorithm, the information of the grading nodes is collected through the second network for voting, and the grading result with the highest score is selected as the final grading result, thereby completing the customized address grading of the business.

[0036] In a second aspect, the present invention provides a custom address processing device based on address structured parsing, the device comprising:

[0037] An acquisition unit, used for acquiring the original address input by the user;

[0038] A search unit connected to the acquisition unit, configured to search the original address through a distributed address index vector library to obtain a matching address having a highest similarity confidence rate with the original address;

[0039] A completion unit, connected to the search unit, for completing the matching address to obtain a target address;

[0040] A parsing unit, connected to the completion unit, for performing structured parsing on the target address to obtain parsed data;

[0041] The grading unit is connected to the parsing unit and is used to perform custom address grading on the service according to the parsing data to complete the custom address processing based on address structured parsing.

[0042] Furthermore, the device also includes a construction unit;

[0043] The construction unit is connected to the search unit and is used to construct a distributed address index vector library;

[0044] The building block comprises:

[0045] The acquisition module is used to obtain OSM address data in the OSM open source address information database;

[0046] A preprocessing module, connected to the acquisition module, for preprocessing the OSM address data to make it conform to the index structure of Elasticsearch;

[0047] The import module is connected to the preprocessing module and is used to import the preprocessed OSM address data into the Elasticsearch index in batches using the Bulk API of Elasticsearch, thereby constructing a distributed address index vector library with multi-mode search capabilities.

[0048] Furthermore, the parsing unit comprises:

[0049] A parsing module, used to parse the target address through a parsing node to obtain a parsing result;

[0050] Wherein, the parsing node is an independent unit or component for parsing the target address; the parsing result is generated by each parsing node receiving the same target address and independently using the Tongyi Qianwen 72B large language model for parsing;

[0051] A first publishing module, connected to the parsing module, for publishing the parsing result in a first network composed of the parsing nodes;

[0052] The first selection module is connected to the first publishing module and is used to adopt the Byzantine Generals Consensus Algorithm to collect information of parsing nodes through the first network for voting, and select the parsing result with the highest score as the final parsing result, thereby completing the generation of parsing data.

[0053] Furthermore, the grading unit comprises:

[0054] A classification module is used to classify services through classification nodes to obtain classification results;

[0055] The grading node is an independent unit or component for grading services; the grading result is obtained by each grading node according to the grading standard and independently using the Tongyi Qianwen 72B large language model to determine the level to which the address in the parsed data belongs; the grading standard is pre-set according to business requirements;

[0056] A second publishing module, connected to the grading module, for publishing the grading result in a second network composed of the grading nodes;

[0057] The second selection module is connected to the second publishing module and is used to adopt the Byzantine Generals Consensus Algorithm to collect information of grading nodes through the second network for voting, and select the grading result with the highest score as the final grading result, thereby completing the customized address grading of the business.

[0058] The present invention can automatically realize address completion and customized address classification, thereby improving the ability, quality and efficiency of address structuring. The specific beneficial effects are as follows:

[0059] 1. Efficient address completion: The present invention can quickly repair missing or incomplete address information, ensure the integrity of address data in special scenarios, and thus improve the user experience.

[0060] 2. Flexible custom level labeling: The present invention can autonomously label the level of address fragments according to different business requirements or data specifications, providing a high degree of flexibility and customizability to meet a variety of application scenarios.

[0061] 3. Automated processing: The entire parsing and annotation process in the present invention does not require human intervention, which significantly improves efficiency and consistency and reduces errors and inconsistencies that may be caused by manual operations.

[0062] 4. Wide adaptability to multiple scenarios: The algorithm of the present invention can adapt to various business scenarios and data specifications, has good adaptability and wide application value, and can be effectively applied in different fields.

[0063] 5. Data consistency: The present invention uses an automated algorithm to ensure that address labeling results in different users or business scenarios are highly consistent, thereby enhancing the reliability and comparability of the data.

[0064] 6. Overall improvement of address structuring capabilities: The present invention can significantly improve capabilities, quality and efficiency in the field of address structuring, and promote the further development of related businesses.

[0065] 7. The present invention can achieve standardized address completion and accurate longitude and latitude acquisition, thereby improving the accuracy and consistency of address information. At the same time, it also provides effective support for geolocation and data analysis, ensuring higher efficiency and reliability in various application scenarios.

[0066] 8. The present invention does not require separate training of AI models, thereby significantly improving the flexibility of the system and enabling it to more conveniently adapt to various application scenarios and needs.

[0067] 9. The present invention can meet diverse data specifications, ensure that the parsing results are widely applicable in different application scenarios, and solve the reliance of address-level labeling on manual processing - this processing method is inefficient and produces inconsistent results. At the same time, it can automatically process the sorting and logical relationships of structured address fragments, significantly improving the efficiency of automatic labeling and translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Schematic diagram of a custom address processing framework based on address structured parsing in an embodiment of the present invention;

[0069] Figure 2 Schematic diagram of a custom address processing method based on address structured parsing in an embodiment of the present invention;

[0070] Figure 3 It is a schematic diagram of the lucene process in an embodiment of the present invention;

[0071] Figure 4 The figure is a schematic diagram of a custom address processing flow based on address structured parsing in an embodiment of the present invention;

[0072] Figure 5 Schematic diagram of a custom address processing device based on address structured parsing in an embodiment of the present invention;

[0073] Among them, the figure marks are: 10, acquisition unit, 20, search unit, 30, completion unit, 40, parsing unit, 50, classification unit. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0075] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0076] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0077] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0078] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0079] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0080] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0081] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0082] Embodiment 1:

[0083] like Figure 1 and Figure 2 As shown, this embodiment provides a custom address processing method based on address structured parsing. The custom address processing method based on address structured parsing provided in this embodiment is widely used in multiple scenarios, including e-commerce logistics to improve delivery accuracy and efficiency, map navigation to optimize routes and location information, customer relationship management to maintain customer information and improve service quality, intelligent customer service to quickly respond to user address inquiries, data analysis and mining to discover market opportunities, administrative management to optimize resource allocation, and banking and financial services to ensure address accuracy, reduce risks and meet regulatory requirements. These application scenarios show that this method can significantly improve the service efficiency and management accuracy of various industries. The method comprises the following steps:

[0084] Step S0: construct a distributed address index vector library;

[0085] The step S0 specifically includes the following steps:

[0086] Get OSM address data from the OSM open source address information database;

[0087] Preprocessing the OSM address data to make it conform to the index structure of Elasticsearch;

[0088] Elasticsearch's Bulk API is used to batch import the pre-processed OSM address data into the Elasticsearch index, thereby building a distributed address index vector library with multi-mode search capabilities. Elasticsearch's Bulk API is an efficient data processing interface that allows users to submit multiple index, update, and delete requests at one time, thereby significantly improving the performance of data operations. By packaging multiple requests into a batch request, BulkAPI reduces the number of network round trips and reduces cluster load, making it more efficient when processing large amounts of data.

[0089] Specifically, the full name of OSM is "OpenStreetMap", which means open street map. It is a free map database where users can freely edit and use map data, covering information such as roads, buildings, and natural landscapes around the world. Elasticsearch is an open source search and data analysis engine based on Apache Lucene. It can provide distributed full-text search, real-time data analysis, and scalable data storage solutions. The OSM address data is preprocessed to conform to the index structure of Elasticsearch, mainly including data cleaning in the existing technology (removing duplicate records, invalid data and data in the wrong format to ensure data accuracy and consistency), field mapping (defining and mapping required fields such as address, city, postal code, etc. according to the index structure requirements of Elasticsearch), format standardization (standardizing address information into a unified format, including unified character encoding, date format, etc.), word segmentation processing (segmenting address fields for subsequent search and query operations), geographic information extraction (extracting and converting geographic coordinates in addresses to support geographic location search and analysis), index optimization (optimizing index settings according to data volume and query requirements, such as setting the number of replicas and shards), data conversion (converting preprocessed data into JSON format supported by Elasticsearch for indexing), etc.

[0090] This embodiment adopts ES distributed search (Elasticsearch, ES is a distributed search engine with powerful distributed search capabilities). The ES distributed search implementation of this embodiment includes the following steps:

[0091] (1) Feature extraction: Use spatial processing and machine learning methods to extract features of map data. For example, use the deep learning model ResNet to extract features such as map name, POI, element label, alias, etc., and encode them into vectors.

[0092] (2) Create an index in Elasticsearch and define appropriate mappings for storing vector data. Use the dense_vector field type of ES to store high-dimensional vector data.

[0093] (3) Store the extracted vector data in the corresponding fields of Elasticsearch. Each data can contain one or more vector fields

[0094] (4) Use the k-NN (k-nearest neighbor) query function of Elasticsearch to perform vector search. By specifying the query vector and search parameters such as k value, num_candidates, etc., find the data most similar to the query vector. The similarity distance uses the dot product algorithm dot_product similarity algorithm that is more in line with the ES architecture.

[0095] For vector A and vector B, the calculation formula of the dot product similarity algorithm is as follows:

[0096]

[0097] Among them, A i represents the i-th element of vector A, B i represents the i-th element of vector B, and n represents the dimension of the vector.

[0098] Step S1: Obtain the original address input by the user.

[0099] Step S2: searching the original address through a distributed address index vector library to obtain a matching address with the highest similarity confidence rate with the original address.

[0100] Specifically, step S2 includes the following steps:

[0101] Based on the Tongyi Qianwen 72B large language model, extract the feature vector corresponding to the original address; use the Bulk API of Elasticsearch to batch import the original address data and the extracted feature vectors into the Elasticsearch index;

[0102] Based on the Elasticsearch index, according to the k-NN (k-Nearest Neighbors, k nearest neighbor algorithm, is a simple and effective machine learning algorithm widely used in classification and regression tasks) query function of Elasticsearch, by specifying the query vector and the search parameter, performing a vector search corresponding to the feature vector and the original address;

[0103] Receive the address data similar to the original address returned by ElasticSearch, and take the address corresponding to the highest similarity confidence rate as the matching address.

[0104] Tongyi Qianwen 72B large language model is an existing model, which refers to an artificial intelligence large language model with 72 billion parameters. It is an advanced artificial intelligence language model with powerful natural language processing capabilities. The model uses large-scale training data and deep learning technology to understand and generate natural language texts, and is widely used in scenarios such as question-answering systems, text generation, and dialogue systems. Its 72B parameter scale enables the model to have strong capabilities in understanding complex contexts, extracting information, and generating high-quality texts, which helps to improve the performance and user experience of various intelligent applications.

[0105] Step S3: Complete the matching address to obtain the target address.

[0106] This step specifically includes: completing the matching address according to the WGS84 longitude and latitude code to obtain the target address.

[0107] The specific steps of completing the matching address according to the WGS84 longitude and latitude code are as follows:

[0108] (1) Input longitude and latitude data: Get the WGS84 longitude and latitude coordinates of the address to be matched.

[0109] (2) Calling geocoding service: Use geocoding API (such as Google Maps API, OpenStreetMap, etc.) to convert longitude and latitude into actual address information. This service can query the relevant address based on the given coordinates.

[0110] (3) Obtaining completed information: Parse the returned address information and extract the required detailed address components, such as province, city, district, street, etc.

[0111] (4) Standardization: The acquired address information is standardized and formatted to meet the system requirements or the user's preset format.

[0112] (5) Output the target address: Output the completed address to ensure that it contains all necessary components to form an accurate and complete target address.

[0113] In addition to the WGS84 longitude and latitude encoding, this embodiment can also use the following method to obtain the complete completion of the target address:

[0114] (1) Geocoding:

[0115] Use the geocoding interface provided by the map service to convert place names, addresses, postal codes and other information into longitude and latitude coordinates, and then use these coordinates to find the complete target address.

[0116] (2) Reverse Geocoding:

[0117] If there is other forms of location information (such as store name, city name, etc.), this information can be converted into longitude and latitude through reverse geocoding services, and then further parsed into a complete address.

[0118] (3) Address resolution API:

[0119] Use a specific address resolution API to directly request complete address information by inputting incomplete address information.

[0120] (4) Map dataset:

[0121] Use open map datasets (such as OpenStreetMap) for queries. These datasets generally provide rich geographic information and can complete addresses based on known partial data.

[0122] (5) User feedback and modifications:

[0123] Allow users to manually enter or correct incomplete address information, and dynamically update and complete address data by collecting user feedback.

[0124] Step S4: Perform structured analysis on the target address to obtain analysis data.

[0125] The step S4 specifically comprises: using the Byzantine Generals Consensus Algorithm to perform structured analysis on the target address to obtain analysis data;

[0126] The Byzantine Generals Consensus Algorithm is used to perform structured analysis on the target address to obtain analysis data, which specifically includes the following steps:

[0127] Parsing the target address through a parsing node to obtain a parsing result;

[0128] Wherein, the parsing node is an independent unit or component for parsing the target address; the parsing result is generated by each parsing node receiving the same target address and independently using the Tongyi Qianwen 72B large language model for parsing;

[0129] Publishing the parsing result in a first network composed of the parsing nodes;

[0130] The Byzantine Generals Consensus Algorithm is adopted to collect information of parsing nodes through the first network for voting, and the parsing result with the highest score is selected as the final parsing result, thereby completing the generation of parsing data.

[0131] The Byzantine Generals Consensus Algorithm is a protocol used to solve the problem of consistency between nodes in a distributed system. It aims to ensure that all loyal nodes can reach a consensus even in the presence of malicious nodes (traitors). The algorithm allows nodes to verify information with each other through multiple rounds of message transmission, thereby achieving consistent decisions and tolerating up to one-third of node failures or faults. The core idea is to exchange redundant information so that even if some nodes spread false messages, the remaining nodes can rely on other information to reach a consensus.

[0132] This embodiment adopts common elements of structured analysis, which are as follows:

[0133]

[0134]

[0135] Based on the above labels, this embodiment uses a large model to label and produce a data set. The labeling results are scored through different platforms to form a consensus to determine whether they are correct, and a training data set is obtained. Finally, a highly efficient structured analysis exclusive model is trained.

[0136] Step S5: According to the parsed data, the services are classified into custom addresses to complete custom address processing based on address structured parsing.

[0137] In step S5, the business is classified by custom address, specifically: the business is classified by custom address using the Byzantine Generals Consensus Algorithm;

[0138] The Byzantine Generals Consensus Algorithm is used to perform custom address classification on the business, which specifically includes the following steps:

[0139] Classify the business through the classification nodes to obtain the classification results;

[0140] Among them, the grading node is an independent unit or component for grading the business; the grading result is obtained by each grading node according to the grading standard and independently using the Tongyi Qianwen 72B large language model to determine the level to which the address in the parsed data belongs; the grading standard is pre-set according to business needs, specifically including: the first grading address is the province, the second grading address is the city, the third grading address is the urban district, county, county-level city, or flag, the fourth grading address is the street, town, township, industrial park, university town, the fifth grading address is the road, street, lane, lane, administrative village; the sixth grading address is the house number, natural village, or natural village group, the seventh grading address is the community, real estate, single building complex, school, natural village homestead, the eighth grading address is the building number, building number, building number, factory number, street, lane and village number of the open address, the ninth grading address is the unit number, the tenth grading address is the floor, and the eleventh grading address is the door number or room number;

[0141] Publishing the grading result in the second network composed of the grading nodes;

[0142] By adopting the Byzantine Generals Consensus Algorithm, the information of the grading nodes is collected through the second network for voting, and the grading result with the highest score is selected as the final grading result, thereby completing the customized address grading of the business.

[0143] For business, different fields and enterprises require different classification rules. For example, the requirements for address level division between the logistics industry and the communications industry are quite different. How to use the structured parsed addresses to efficiently and flexibly support the classification needs of different industries is a major feature of this embodiment.

[0144] This embodiment only requires the user to provide classification rules, some structured parsed data and corresponding classification data, and uses a large model for self-supervision to obtain a data set, and then trains a dedicated classification model for use. Taking a certain business scenario as an example, the eleven classification rules are required as follows:

[0145] First level: Province;

[0146] Second level: city;

[0147] The third level: municipal districts, counties, county-level cities, and banners;

[0148] Fourth level: streets, towns, townships, industrial parks, and university towns;

[0149] Fifth level: roads, streets, lanes, alleys, and administrative villages;

[0150] Level 6: house number, natural village + village group;

[0151] Level 7: residential areas, real estate projects, building complexes (including bungalows and communities), schools, and natural village housing sites;

[0152] Level 8: Building number, block number, structure number, factory number, street, lane, and village number of the open address;

[0153] Level 9: Unit;

[0154] Level 10: Floor;

[0155] Level 11: Room (Door Number).

[0156] like Figure 4 As shown, this embodiment is based on the 72B model (the 72B model refers to an artificial intelligence large language model with a parameter amount of 72 billion) to process the input address "XX Province XX City XX District XX Town XX Village XX District". First, the address is annotated using the BIESO annotation method; then, information is extracted through the 72B large model, using prompt words such as role, rule, input, output, and format for guidance. Next, the data will enter the voting model cluster, including large model A, large model B, and large model C, and after consensus is reached, a labeled data set will be generated. Subsequently, the data set is loaded for model training to obtain a base training model based on BERT, and finally the construction of a structured parsing exclusive model is completed through an exclusive model.

[0157] The overall process scheme of this embodiment is as follows:

[0158] 1) Obtain the OSM open source address information database.

[0159] 2) Establish an address index vector library through the ES distributed search engine to provide multi-mode search capabilities.

[0160] 3) Search the original address through ES distributed search and obtain the result with the highest confidence rate in the address vector coordinate system.

[0161] 4) Read the WGS84 longitude and latitude code of the result.

[0162] 5) Through ES distributed search for longitude and latitude codes, the complete address with the highest confidence rate in the address database is obtained to complete the completion.

[0163] 6) Given a number of structured parsing graded samples, compile prompt words, call the large model to learn, autonomously generate samples, iterate learning again, and output the structured parsing results.

[0164] 7) Use the multi-mode instance "Byzantine Generals Consensus Algorithm" to vote on the output results, automatically supervise and correct the learning and training of large models.

[0165] Repeat steps 6) and 7) above to complete the autonomous learning and processing of business classification.

[0166] like Figure 3As shown, this embodiment uses Lucene to implement the data flow. Specifically, the flow chart is divided into two parts, one part involves index construction, and the other part involves query processing. The indexing and querying process of Lucene will be explained in detail below.

[0167] 1. Index building process:

[0168] (1) Files to be indexed (document input):

[0169] Lucene first receives documents to be indexed, which usually appear in the form of text. Documents can be various data sources such as web pages, books, and articles.

[0170] (2) Lexical analysis and language processing:

[0171] After the initial processing, the document will enter the lexical analysis and language processing stage. This process includes breaking down the document into terms, for example, removing stop words (such as "the" and "and"), performing stemming, and normalizing words.

[0172] (3) Index creation:

[0173] The text that has been processed by lexical analysis will be converted into Lucene's inverted index structure. This index associates the terms in the document with their positions in the document to form a data structure that can be searched quickly. The inverted index is usually stored in a mapping method of "term to document ID".

[0174] (4) Index storage:

[0175] The created index will be written to disk or other persistent storage. This is to ensure that the index can be stored efficiently and retrieved quickly during subsequent queries.

[0176] The index storage module is responsible for managing the storage of these index data and ensuring fast access during query.

[0177] Query processing:

[0178] a. Query statement input:

[0179] The user submits a query statement, which may be a natural language text (such as "find articles related to a certain topic"), and Lucene will parse the user's query.

[0180] b. Lexical analysis:

[0181] Lucene performs the same lexical analysis on the query as it does on documents during indexing. The purpose of this step is to break the query into standardized terms that can be matched against the data in the index.

[0182] c. Syntax analysis:

[0183] The query is further parsed to determine the weight and relevance of each term in the query.

[0184] d. Index search:

[0185] Once the query is parsed and converted into terms, Lucene searches for documents that match the query terms in the pre-built index library. This process uses the structure of the inverted index to quickly locate documents containing the query terms.

[0186] e. Relevance ranking:

[0187] Lucene calculates a relevance score for each matching document (usually based on the TF-IDF algorithm or BM25). This score reflects how relevant each document is to the query. The higher the score, the more closely the document matches the query.

[0188] f. Output of relevance sorting:

[0189] According to the relevance score, Lucene sorts all matching documents and returns these sorted results to the user. In the end, the user sees the documents that are most relevant to the query.

[0190] g. Query result output:

[0191] Users end up with a ranked list of documents that shows the results most relevant to their query.

[0192] This example is implemented through code (code omitted, there is code and it has been encapsulated into an interface). The code of this embodiment is based on the FastAPI framework and creates an API service for processing address resolution. The main function of the code is to handle address completion, address structuring and custom level annotation. It mainly includes the following contents:

[0193] (1) Log Recording: The log module records important information during the server operation, such as request processing status, error information, etc.

[0194] (2) Address standardization interface ( / standardaddr):

[0195] The address and city information submitted by the user will be passed into the system and first completed (via complete.process).

[0196] If the completion is successful, the completed address will be structured translated (via transfer.process).

[0197] Returns standardized address information, including address classification (from province to floor, room number) and address latitude and longitude information.

[0198] (3) Alias ​​processing interface ( / alias):

[0199] Receive the queried address alias, call alias.process to process it, and then return the relevant address data.

[0200] (4) Error handling: The code includes detailed exception capture and error logging for each step, ensuring that appropriate HTTP error codes are returned and the cause of the error is recorded when an exception occurs.

[0201] (5) Automated processing: The code calls external services to complete addresses and perform structured processing. It can also automatically perform hierarchical labeling of addresses, eliminating manual intervention and improving processing efficiency.

[0202] Receive address-related requests through code, complete address completion, structured parsing, and output address information by level. It also has a flexible error handling mechanism and automated process.

[0203] This embodiment can automatically complete addresses and customize address classification, thereby improving the ability, quality and efficiency of address structuring. This embodiment can support automatic custom level annotation according to business needs or data specifications, thereby providing a high degree of flexibility and customizability. In addition, the entire parsing and annotation process is automated, eliminating manual intervention, improving efficiency and consistency, and ensuring that the address annotation results for different users or scenarios remain highly consistent, which has a wide range of application value.

[0204] Embodiment 2:

[0205] like Figure 5 As shown, this embodiment provides a custom address processing device based on address structured parsing, the device comprising:

[0206] The acquisition unit 10 is used to acquire the original address input by the user;

[0207] A search unit 20, connected to the acquisition unit 10, is used to search the original address through a distributed address index vector library to obtain a matching address with a highest similarity confidence rate with the original address;

[0208] A completion unit 30, connected to the search unit 20, for completing the matching address to obtain a target address;

[0209] The parsing unit 40 is connected to the completion unit 30 and is used to perform structured parsing on the target address to obtain parsed data;

[0210] The classification unit 50 is connected to the parsing unit 40 and is used to perform custom address classification on the service according to the parsing data to complete the custom address processing based on the address structured parsing.

[0211] As a specific embodiment, the device further includes a construction unit;

[0212] The construction unit is connected to the search unit and is used to construct a distributed address index vector library;

[0213] The building block comprises:

[0214] The acquisition module is used to obtain OSM address data in the OSM open source address information database;

[0215] A preprocessing module, connected to the acquisition module, for preprocessing the OSM address data to make it conform to the index structure of Elasticsearch;

[0216] The import module is connected to the preprocessing module and is used to import the preprocessed OSM address data into the Elasticsearch index in batches using the Bulk API of Elasticsearch, thereby constructing a distributed address index vector library with multi-mode search capabilities.

[0217] As a specific implementation, the parsing unit 40 includes:

[0218] A parsing module, used to parse the target address through a parsing node to obtain a parsing result;

[0219] Wherein, the parsing node is an independent unit or component for parsing the target address; the parsing result is generated by each parsing node receiving the same target address and independently using the Tongyi Qianwen 72B large language model for parsing;

[0220] A first publishing module, connected to the parsing module, for publishing the parsing result in a first network composed of the parsing nodes;

[0221] The first selection module is connected to the first publishing module and is used to adopt the Byzantine Generals Consensus Algorithm to collect information of parsing nodes through the first network for voting, and select the parsing result with the highest score as the final parsing result, thereby completing the generation of parsing data.

[0222] As a specific implementation, the classification unit 50 includes:

[0223] A classification module is used to classify services through classification nodes to obtain classification results;

[0224] Among them, the grading node is an independent unit or component for grading the business; the grading result is obtained by each grading node according to the grading standard and independently using the Tongyi Qianwen 72B large language model to determine the level to which the address in the parsed data belongs; the grading standard is pre-set according to business needs, specifically including: the first grading address is the province, the second grading address is the city, the third grading address is the urban district, county, county-level city, or flag, the fourth grading address is the street, town, township, industrial park, university town, the fifth grading address is the road, street, lane, lane, administrative village; the sixth grading address is the house number, natural village, or natural village group, the seventh grading address is the community, real estate, single building complex, school, natural village homestead, the eighth grading address is the building number, building number, building number, factory number, street, lane and village number of the open address, the ninth grading address is the unit number, the tenth grading address is the floor, and the eleventh grading address is the door number or room number;

[0225] A second publishing module, connected to the grading module, for publishing the grading result in a second network composed of the grading nodes;

[0226] The second selection module is connected to the second publishing module and is used to adopt the Byzantine Generals Consensus Algorithm to collect information of grading nodes through the second network for voting, and select the grading result with the highest score as the final grading result, thereby completing the customized address grading of the business.

[0227] The device in this embodiment can execute the method in embodiment 1.

[0228] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for processing a custom address based on structured address parsing, characterized in that: The method comprises the following steps: Step S1: Obtain the original address input by the user; Step S2: searching the original address through a distributed address index vector library to obtain a matching address with the highest similarity confidence rate with the original address; Step S3: completing the matching address to obtain a target address; Step S4: performing structural analysis on the target address to obtain analysis data; Step S5: According to the parsed data, the services are classified into custom addresses to complete custom address processing based on address structured parsing.

2. The method for processing a custom address based on structured address analysis according to claim 1, characterized in that: Before step S2, the method further includes step S0; Step S0: construct a distributed address index vector library; The step S0 specifically includes the following steps: Get OSM address data from the OSM open source address information database; Preprocessing the OSM address data to make it conform to the index structure of Elasticsearch; The preprocessed OSM address data is imported into the Elasticsearch index in batches using the Elasticsearch Bulk API, thereby building a distributed address index vector library with multi-mode search capabilities.

3. The method for processing a custom address based on structured address analysis according to claim 1, characterized in that: The step S2 specifically includes the following steps: Based on the Tongyi Qianwen 72B large language model, extracting a feature vector corresponding to the original address; Use Elasticsearch's Bulk API to batch import the original address data and extracted feature vectors into the Elasticsearch index; Based on the Elasticsearch index, according to the k-NN query function of Elasticsearch, by specifying the query vector and the search parameter, performing a vector search corresponding to the feature vector and the original address; Receive the address data similar to the original address returned by ElasticSearch, and take the address corresponding to the highest similarity confidence rate as the matching address.

4. The method for processing a custom address based on structured address analysis according to claim 1, characterized in that: The step S3 is specifically: The matching address is fully completed according to the WGS84 longitude and latitude code to obtain the target address.

5. The method for processing a custom address based on structured address analysis according to claim 1, characterized in that: The step S4 specifically comprises: using the Byzantine Generals Consensus Algorithm to perform structured analysis on the target address to obtain analysis data; The Byzantine Generals Consensus Algorithm is used to perform structured analysis on the target address to obtain analysis data, which specifically includes the following steps: Parsing the target address through a parsing node to obtain a parsing result; Wherein, the parsing node is an independent unit or component for parsing the target address; the parsing result is generated by each parsing node receiving the same target address and independently using the Tongyi Qianwen 72B large language model for parsing; Publishing the parsing result in a first network composed of the parsing nodes; The Byzantine Generals Consensus Algorithm is adopted to collect information of parsing nodes through the first network for voting, and the parsing result with the highest score is selected as the final parsing result, thereby completing the generation of parsing data.

6. The method for processing a custom address based on structured address analysis according to any one of claims 1 to 5, characterized in that: In the step S5, the business is classified by custom address, specifically: the business is classified by custom address using the Byzantine Generals Consensus Algorithm; The Byzantine Generals Consensus Algorithm is used to perform custom address classification on the business, which specifically includes the following steps: Classify the business through the classification nodes to obtain the classification results; The grading node is an independent unit or component for grading services; the grading result is obtained by each grading node according to the grading standard and independently using the Tongyi Qianwen 72B large language model to determine the level to which the address in the parsed data belongs; the grading standard is pre-set according to business requirements; Publishing the grading result in the second network composed of the grading nodes; By adopting the Byzantine Generals Consensus Algorithm, the information of the grading nodes is collected through the second network for voting, and the grading result with the highest score is selected as the final grading result, thereby completing the customized address grading of the business.

7. A custom address processing device based on address structured parsing, characterized in that: The device comprises: An acquisition unit, used for acquiring the original address input by the user; A search unit connected to the acquisition unit, configured to search the original address through a distributed address index vector library to obtain a matching address having a highest similarity confidence rate with the original address; A completion unit, connected to the search unit, for completing the matching address to obtain a target address; A parsing unit, connected to the completion unit, for performing structured parsing on the target address to obtain parsed data; The grading unit is connected to the parsing unit and is used to perform custom address grading on the service according to the parsing data to complete the custom address processing based on address structured parsing.

8. The device for processing custom addresses based on address structured parsing according to claim 7, characterized in that: The device also includes a construction unit; The construction unit is connected to the search unit and is used to construct a distributed address index vector library; The building block comprises: The acquisition module is used to obtain OSM address data in the OSM open source address information database; A preprocessing module, connected to the acquisition module, for preprocessing the OSM address data to make it conform to the index structure of Elasticsearch; The import module is connected to the preprocessing module and is used to import the preprocessed OSM address data into the Elasticsearch index in batches using the Bulk API of Elasticsearch, thereby constructing a distributed address index vector library with multi-mode search capabilities.

9. The device for processing custom addresses based on structured address analysis according to claim 7, characterized in that: The parsing unit comprises: A parsing module, used to parse the target address through a parsing node to obtain a parsing result; Wherein, the parsing node is an independent unit or component for parsing the target address; the parsing result is generated by each parsing node receiving the same target address and independently using the Tongyi Qianwen 72B large language model for parsing; A first publishing module, connected to the parsing module, for publishing the parsing result in a first network composed of the parsing nodes; The first selection module is connected to the first publishing module and is used to adopt the Byzantine Generals Consensus Algorithm to collect information of parsing nodes through the first network for voting, and select the parsing result with the highest score as the final parsing result, thereby completing the generation of parsing data.

10. The device for processing a custom address based on structured address analysis according to any one of claims 7 to 9, characterized in that: The grading unit comprises: A classification module is used to classify services through classification nodes to obtain classification results; The grading node is an independent unit or component for grading services; the grading result is obtained by each grading node according to the grading standard and independently using the Tongyi Qianwen 72B large language model to determine the level to which the address in the parsed data belongs; the grading standard is pre-set according to business requirements; A second publishing module, connected to the grading module, for publishing the grading result in a second network composed of the grading nodes; The second selection module is connected to the second publishing module and is used to adopt the Byzantine Generals Consensus Algorithm to collect information of grading nodes through the second network for voting, and select the grading result with the highest score as the final grading result, thereby completing the customized address grading of the business.

Images