RAG-based address information standardization processing method and device

Abstract

The invention discloses an RAG-based address information standardization processing method and apparatus. The method comprises the steps of determining first address information in text description information based on a large model; determining a first address vector corresponding to the first address information based on the RAG; determining a first similarity between the first address vector and at least one standard address vector in a standard address base, and determining at least one second address vector matched with the first address vector according to the first similarity; generating a cue word according to standard address information corresponding to at least one second address vector in a standard address library, and determining the second address information in the text description information based on the cue word and the large model; determining the semantic similarity between the second address information and at least one piece of address information with complete hierarchy, and determining third address information matched with the second address information according to the semantic similarity; and performing hierarchical completion on the second address information according to the third address information to obtain the target address information, thereby providing preconditions for accurate intelligence analysis.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of intelligent traffic management, and in particular to an address information standardization processing method and device based on RAG. BACKGROUND

[0002] Intelligence analysis using address information refers to a process of extracting valuable intelligence related to geographical position, user behavior and event mode through address data collection, processing, correlation and mining, and providing support for decision-making.

[0003] Due to the non-standard problems of the input address information, such as missing address levels, the address levels including province, city, district, street, road and POI (Point of Interest), in actual intelligence analysis scenarios, the non-standard problems of the address information will lead to inaccurate intelligence analysis. SUMMARY

[0004] The application provides an address information standardization processing method and device based on RAG, to solve the problem of inaccurate intelligence analysis caused by non-standard address information.

[0005] In a first aspect, the application provides an address information standardization processing method based on RAG, which comprises the following steps:

[0006] determining first address information to be processed in text description information based on a large model, determining a first address vector corresponding to the first address information based on RAG (Retrieval Augmentation Generation), determining a first similarity between the first address vector and at least one standard address vector in a standard address library, determining at least one second address vector in the standard address library matching the first address vector according to the first similarity, generating a prompt word according to standard address information corresponding to the at least one second address vector in the standard address library, and determining second address information in the text description information based on the prompt word and the large model;

[0007] determining a semantic similarity between the second address information and at least one address information with complete levels pre-stored, determining third address information matching the second address information according to the semantic similarity, and performing level completion on the second address information according to the third address information to obtain target address information.

[0008] The above technical solution has the following advantages or beneficial effects:

[0009] In the field of intelligent traffic management technology, the traffic management department usually uses address information for intelligence analysis. The standardization and accuracy of address information are the premise of the accuracy of intelligence analysis. In related technologies, intelligence analysis is generally directly performed according to the input address information. However, the input address information may have the problem of incomplete address information due to missing levels, which may affect the accuracy of intelligence analysis based on address information. Based on the above consideration, the present application proposes a RAG-based address information standardization processing method, which aims to standardize the address information and provide a prerequisite for accurate intelligence analysis. Specifically, first, the first address information in the text description information is determined based on a large model, and the first address information is converted into a first address vector based on the retrieval enhancement generation technology RAG. Then, the second address vector matched with the first address vector is determined according to the first similarity between the first address vector and the standard address vector in the standard address library. Further, the prompt word is generated according to the standard address information corresponding to the second address vector, and the second address information in the text description information is obtained based on the prompt word. The present application performs intelligent analysis of address information based on RAG and the standard address library, which can realize address key information extraction and intelligent word segmentation, improve the accuracy of address analysis, and thus make the obtained second address information more accurate than the first address information. Further, after determining the second address information, the third address information matched with the second address information is determined according to the semantic similarity between the second address information and the pre-stored level-complete address information, and then the second address information is supplemented with levels according to the third address information to obtain target address information. Thus, the level completion of the second address information is realized, and the standardized and complete target address information is obtained, which provides a prerequisite for accurate intelligence analysis.

[0010] In a second aspect, the present application provides a RAG-based address information standardization processing device, which comprises:

[0011] A first determination module is configured to determine first address information to be processed in text description information based on a large model, determine a first address vector corresponding to the first address information based on retrieval enhancement generation technology RAG, determine a first similarity between the first address vector and at least one standard address vector in a standard address library, determine at least one second address vector in the standard address library matched with the first address vector according to the first similarity, generate a prompt word according to standard address information corresponding to the at least one second address vector in the standard address library, and determine second address information in the text description information based on the prompt word and the large model.

[0012] The second determining module is configured to determine semantic similarity between the second address information and pre-stored hierarchical complete address information, determine third address information matched with the second address information according to the semantic similarity, and perform hierarchical completion on the second address information according to the third address information to obtain target address information.

[0013] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0014] The memory is configured to store a computer program.

[0015] The processor is configured to execute the program stored on the memory to implement the method.

[0016] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method.

[0017] In a fifth aspect, the present application provides a computer program product, wherein the computer program product comprises an executable program, and the executable program is executed by a processor to implement the method. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 A schematic diagram of the address information standardization processing process based on RAG provided by the present application is shown in the figure.

[0020] Figure 2 A schematic diagram of the process of determining at least one second address vector matched with the first address vector in the standard address library provided by the present application is shown in the figure.

[0021] Figure 3 A schematic diagram of the process of determining target address information provided by the present application is shown in the figure.

[0022] Figure 4 A schematic diagram of the homonym replacement process provided by the present application is shown in the figure.

[0023] Figure 5 A schematic diagram of the address information processing process provided by the present application is shown in the figure.

[0024] Figure 6Standardization flow chart of address information provided for the present application;

[0025] Figure 7 Detailed flow chart of RAG-based address information standardization processing provided for the present application;

[0026] Figure 8 Schematic diagram of clustering effect provided for the present application;

[0027] Figure 9 Structure schematic diagram of RAG-based address information standardization processing device provided for the present application;

[0028] Figure 10 Structure schematic diagram of electronic device provided for the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose and implementation of the present application more clear, the following will combine the drawings in the exemplary embodiments of the present application to clearly and completely describe the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only some of the embodiments of the present application, but not all the embodiments.

[0030] It should be noted that the brief description of the terms in the present application is only for the convenience of understanding the following described embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0031] The terms "first", "second", "third" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit the specific order or sequence, unless otherwise specified. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0032] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not necessarily limit to all the components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0033] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware or / and software code capable of performing functions related to the element.

[0034] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0035] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to exhaust or limit the embodiments to the specific forms disclosed above. Various modifications and variations can be derived according to the above teachings. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different modified embodiments suitable for specific use considerations.

[0036] Figure 1 The RAG-based address information standardization processing process provided in the present application is shown in the figure, which includes the following steps:

[0037] S101: determining first address information to be processed in the text description information based on a large model; determining a first address vector corresponding to the first address information based on retrieval enhancement generation technology RAG; determining a first similarity between the first address vector and at least one standard address vector in a standard address library, and determining at least one second address vector in the standard address library that matches the first address vector according to the first similarity; generating a prompt word based on the standard address information corresponding to the at least one second address vector in the standard address library, and determining second address information in the text description information based on the prompt word and the large model;

[0038] S102: determining the semantic similarity between the second address information and at least one pre-saved hierarchical complete address information, and determining third address information that matches the second address information according to the semantic similarity; performing hierarchical completion on the second address information according to the third address information to obtain target address information.

[0039] The RAG-based address information standardization processing method provided in the present application is applied to an electronic device, which can be a PC, a computer, a smart terminal, a server, etc.

[0040] The electronic device first acquires the input text description information containing the address, then inputs the text description information and the prompt word for extracting the address information in the text description information into the large model, determines the address information in the text description information based on the large model, and the address information is the first address information to be processed. After determining the first address information, the first address information is vectorized based on the retrieval enhancement generation technology RAG to obtain the first address vector corresponding to the determined first address information.

[0041] The standard address library is pre-stored in the electronic device, and the standard address library includes standard address information and a standard address vector corresponding to the standard address information. The standard address vector corresponding to the standard address information is obtained by vectorizing the standard address information based on RAG. The first similarity between the first address vector and at least one standard address vector in the standard address library is determined, and at least one second address vector in the standard address library that matches the first address vector is determined according to the first similarity. Optionally, the first similarity between the first address vector and at least one standard address vector in the standard address library can be determined by an algorithm such as Euclidean distance, cosine similarity, etc. After determining at least one first similarity, the first similarities are sorted in descending order, and a preset number of second address vectors with larger first similarities are selected as the second address vectors in the standard address library that match the first address vector. The preset number is at least 1. After determining at least one second address vector that matches the first address vector, the standard address information corresponding to the at least one second address vector is determined, and then a prompt word is generated according to the determined at least one standard address information. Optionally, the prompt word includes the determined at least one standard address information and a description word for extracting the address information in the text description information.

[0042] The prompt word generated according to the standard address information corresponding to at least one second address vector in the standard address library and the text description information are input into the large model, and the second address information in the text description information is determined based on the large model. The hierarchical complete multiple standard address libraries are pre-stored in the electronic device. For example, a corresponding standard address library can be saved at the municipal level, such as a hierarchical complete standard address library A corresponding to A city, a hierarchical complete standard address library B corresponding to B city, etc. Each hierarchical complete standard address library includes at least one hierarchical complete address information corresponding to the municipal level.

[0043] After the second address information is determined, first, according to the "city level" information in the second address information, a corresponding hierarchical complete standard address library is determined, for example, the "city level" information in the second address information is A city, and then the corresponding hierarchical complete standard address library is determined as the standard address library A of A city. Then, the semantic similarity of the second address information and at least one address information in the corresponding hierarchical complete standard address library is determined. Optionally, the above semantic similarity can be determined by searching the relevance score BM25 algorithm, the Lesk algorithm for word sense disambiguation similarity calculation, the text edit distance algorithm, etc. According to the semantic similarity, the third address information matched with the second address information is determined. Specifically, according to the semantic similarity, the third address information matched with the second address information is determined, including: taking the hierarchical complete address information corresponding to the maximum semantic similarity as the third address information matched with the second address information. According to the third address information, the hierarchical completion of the second address information is performed, and the target address information is obtained. Generally, it is necessary to determine which level information is missing in the second address information, and then the corresponding level information in the third address information is supplemented into the second address information to obtain the target address information.

[0044] In the field of intelligent traffic management technology, the traffic management department usually uses address information for intelligence analysis. The standardization and accuracy of address information are the premise of the accuracy of intelligence analysis. In related technologies, intelligence analysis is generally directly performed according to the input address information. However, the input address information may have the problem of incomplete address information due to missing levels, which may affect the accuracy of intelligence analysis based on address information. Based on the above consideration, the present application proposes an address information standardization processing method based on RAG, which aims to standardize the address information and provide a prerequisite for accurate intelligence analysis. Specifically, first, the first address information in the text description information is determined based on a large model, and the first address information is converted into a first address vector based on the retrieval enhancement generation technology RAG. Then, the second address vector matching the first address vector is determined according to the first similarity between the first address vector and the standard address vector in the standard address library. Further, the second address information in the text description information is obtained based on the prompt word generated according to the standard address information corresponding to the second address vector. The present application uses RAG and the standard address library to intelligently analyze the address information, which can realize address key information extraction and intelligent word segmentation, improve the accuracy of address analysis, and thus make the obtained second address information more accurate than the first address information. Further, after determining the second address information, the third address information matching the second address information is determined according to the semantic similarity between the second address information and the pre-stored level-complete address information, and then the second address information is supplemented with levels according to the third address information to obtain target address information. Thus, the level completion of the second address information is realized, and the standardized and complete target address information is obtained, which provides a prerequisite for accurate intelligence analysis.

[0045] Figure 2 The process provided by the present application for determining at least one second address vector matching the first address vector in the standard address library, determining the first similarity between the first address vector and at least one standard address vector in the standard address library, and determining at least one second address vector matching the first address vector in the standard address library according to the first similarity comprises:

[0046] S201: Determine the first sub-similarity between the first address vector and at least one road standard address vector in the road standard address library, and select at least one road standard address vector as at least one second address vector matching the first address vector in the road standard address library in the order of the first sub-similarity from large to small;

[0047] S202: determine a second sub-similarity between the first address vector and at least one street standard address vector in the street standard address library, and select at least one street standard address vector in descending order of the second sub-similarity as at least one second address vector in the street standard address library matching the first address vector;

[0048] S203: determine a third sub-similarity between the first address vector and at least one POI standard address vector in the POI standard address library, and select at least one POI standard address vector in descending order of the third sub-similarity as at least one second address vector in the POI standard address library matching the first address vector.

[0049] In this application, the standard address library includes a road standard address library, a street standard address library and a POI standard address library. The road standard address library includes address information at the road level and corresponding standard address vectors; the street standard address library includes address information at the street level and corresponding standard address vectors; and the POI standard address library includes address information at the POI level and corresponding standard address vectors.

[0050] Preferably, the first sub-similarity between the first address vector and each road standard address vector in the road standard address library is determined, and a first number of road standard address vectors are selected in descending order of the first sub-similarity as second address vectors in the road standard address library matching the first address vector. The first number is, for example, 4, 5, 6, etc. The second sub-similarity between the first address vector and each street standard address vector in the street standard address library is determined, and a second number of street standard address vectors are selected in descending order of the second sub-similarity as second address vectors in the street standard address library matching the first address vector; the second number can be the same as or different from the first number. The third sub-similarity between the first address vector and each POI standard address vector in the POI standard address library is determined, and a third number of POI standard address vectors are selected in descending order of the third sub-similarity as second address vectors in the POI standard address library matching the first address vector; the third number can be the same as or different from the first number.

[0051] Figure 3 The process for determining target address information provided in this application includes:

[0052] S301: determine first address token information of the second address information, and determine a first level to be completed in the second address information according to the level corresponding to the first address token information;

[0053] S302: Determine the second address segmentation information of the third address information, determine the second address segmentation information corresponding to the first level according to the level corresponding to the second address segmentation information; complete the level of the second address information according to the second address segmentation information corresponding to the first level, and obtain the target address information.

[0054] In this application, first, the second address information is processed by segmentation to obtain each first address segmentation information in the second address information, and the level corresponding to each first address segmentation information is determined respectively, and then the first level to be completed in the second address information can be determined. Then, the third address information is processed by segmentation to obtain each second address segmentation information in the third address information, and the second address segmentation information corresponding to the first level in each second address segmentation information is determined respectively; finally, the second address information is completed according to the second address segmentation information corresponding to the first level in each second address segmentation information, and the target address information is obtained.

[0055] For example, the second address information "traffic management bureau" is completed to obtain the target address information "certain district certain road public security bureau traffic management bureau".

[0056] Figure 4 The same sound word replacement process provided in this application is shown in the figure, after determining the second address information in the text description information based on the prompt word and the large model, before determining the semantic similarity between the second address information and the pre-stored at least one address information with complete level, the method further comprises:

[0057] S401: Determine at least one third address segmentation information of the second address information; wherein the third address segmentation information includes at least one of road address segmentation information, street address segmentation information and POI address segmentation information;

[0058] S402: For the at least one third address segmentation information, determine at least one standard address information in the standard address library corresponding to the third address segmentation information which is homophonic with the third address segmentation information; determine the text similarity between the third address segmentation information and the at least one standard address information by text distance matching algorithm; if the maximum text similarity is greater than the preset similarity threshold, the standard address information corresponding to the maximum text similarity is used to replace the third address segmentation information.

[0059] In the same sound word replacement process, the second address information is processed by segmentation to obtain each segmentation information in the second address information, which is called third address segmentation information.

[0060] If the third address segmentation information is road address segmentation information, at least one road standard address information homonymic with the third address segmentation information is determined in a road standard address library, and a text similarity between the third address segmentation information and the at least one road standard address information is determined through a text distance matching algorithm; if the maximum text similarity is greater than a preset similarity threshold, a road standard address information corresponding to the maximum text similarity is used to replace the third address segmentation information.

[0061] If the third address segmentation information is street address segmentation information, at least one street standard address information homonymic with the third address segmentation information is determined in a street standard address library, and a text similarity between the third address segmentation information and the at least one street standard address information is determined through a text distance matching algorithm; if the maximum text similarity is greater than a preset similarity threshold, a street standard address information corresponding to the maximum text similarity is used to replace the third address segmentation information.

[0062] If the third address segmentation information is POI address segmentation information, at least one POI standard address information homonymic with the third address segmentation information is determined in a POI standard address library, and a text similarity between the third address segmentation information and the at least one POI standard address information is determined through a text distance matching algorithm; if the maximum text similarity is greater than a preset similarity threshold, a POI standard address information corresponding to the maximum text similarity is used to replace the third address segmentation information.

[0063] Figure 5 The address information processing process provided in the application, after the third address segmentation information is replaced by the standard address information corresponding to the maximum text similarity, before the semantic similarity between the second address information and the at least one pre-stored hierarchical complete address information is determined, the method further comprises:

[0064] S501: The second address information obtained after the third address segmentation information is replaced is matched with pre-stored non-address information to determine and delete non-address information in the second address information.

[0065] S502: Repetitive address information in the second address information is identified, and the repetitive address information is de-duplicated.

[0066] S503: The road number in the second address information is mapped to a standard road number range through a preset road number mapping rule.

[0067] S504: For the at least one third address segmentation information, if the third address segmentation information exists in a pre-stored synonym library, a synonymous standard address information corresponding to the third address segmentation information in the synonym library is used to replace the third address segmentation information.

[0068] Before determining the semantic similarity between the second address information and the pre-stored hierarchical complete address information, the second address information is subjected to stop word filtering, de-duplication, road number mapping, and synonym replacement.

[0069] The stop word filtering process is as follows: the non-address information (or stop words) is pre-stored, the second address information obtained after replacing the third address segmentation information is matched with the pre-stored non-address information, the non-address information in the second address information is determined, and then the non-address information in the second address information is deleted, thereby achieving stop word filtering.

[0070] The de-duplication process is as follows: the repeated address information in the second address information obtained after replacing the third address segmentation information is identified, for example, “A Road A Road” is repeated, and then the repeated address information is subjected to de-duplication processing, only one address information of the repeated address information is retained, thereby achieving de-duplication. That is, “A Road A Road” is de-duplicated to “A Road”.

[0071] The road number mapping process is as follows: through a pre-set road number mapping rule, the road number in the second address information is mapped to a standard road number range, for example, the standard road number range specified in the pre-set road number mapping rule is 1 to 10, 11 to 20. If the road number in the second address information is 5, then “5 Road” in the second address information is modified to “1 to 10 Road”. If the road number in the second address information is 12, then “12 Road” in the second address information is modified to “11 to 20 Road”.

[0072] The synonym replacement process is as follows: a synonym library is pre-stored, and the synonym library includes standard address information and corresponding address information. For example, “Traffic Management Bureau”, “Traffic Management Bureau”, and “District Road 2” are the same address, and “Traffic Management Bureau”, “Traffic Management Bureau”, and “District Road 2” are synonyms, wherein “Traffic Management Bureau” is the standard address information, and “Traffic Management Bureau” and “District Road 2” are the synonymous address information of “Traffic Management Bureau”.

[0073] In the present application, after obtaining the target address information, the method further includes:

[0074] The DBSCAN semantic clustering algorithm is used to cluster at least one address information in the address information library containing the target address information, to obtain at least one cluster; and a hot address area is determined according to the number of address information contained in the at least one cluster.

[0075] The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is applied to at least one address information in the address information database to obtain at least one cluster. Then, the number of address information included in the at least one cluster is determined, and a hot address area is determined according to the number of address information included in the at least one cluster. Optionally, a cluster with the largest number of address information is determined, and an area including address information in the cluster is determined as the hot address area. Alternatively, a cluster with the largest number of address information is determined, and if the number of address information included in the cluster is greater than a preset number threshold, an area including address information in the cluster is determined as the hot address area. Alternatively, clusters with a number of address information greater than a preset number threshold are all determined as aggregation clusters, and an area including address information in the aggregation clusters is determined as the hot address area.

[0076] In the present application, after obtaining the at least one cluster, before determining the hot address area according to the number of address information included in the at least one cluster, the method further comprises:

[0077] For at least one outlier obtained after the clustering process, other address information points in a preset area including the outlier are determined, if the other address information points include a non-outlier, the outlier is attributed to a cluster in which the non-outlier is located, if the other address information points are all outliers, the outlier and the other address information are aggregated into one cluster. It should be noted that if there is no other address information point in the preset area including the outlier, the outlier is determined as a real outlier.

[0078] The present application belongs to the field of intelligent traffic management, and relates to address multi-scale analysis, address standardization, spatial semantic analysis, address hotspot analysis based on retrieval augmented generation (RAG).

[0079] In actual intelligence analysis scenarios, address texts often have problems such as expression diversity, incomplete hierarchical information, address record misspelling, and address errors. Traditional address clustering algorithms are often based on single address or coordinate information analysis, and the analysis effect is not good without address standardization and clustering algorithm enhancement.

[0080] The present application breaks through from two dimensions of data enhancement and algorithm optimization, and proposes an address standardization and hotspot mining algorithm based on RAG and spatial semantic calculation, which specifically includes the following core technologies:

[0081] 1. RAG-based address text intelligent parsing: relying on administrative division, road, POI, and other standard address corpus, a structured address knowledge base is constructed, and RAG (retrieval augmentation generation) technology is used to realize address key information extraction and intelligent word segmentation, thereby improving the accuracy of address parsing.

[0082] 2. Multi-scale address standardization and error correction: through stop word filtering, word de-duplication, address number mapping, and other technologies, the normalization expression of address text is optimized. Combined with homophone conversion, synonym replacement, and context completion, the problems of wrong characters, missing levels, and ambiguous expressions are repaired, and multi-level address standardization of province-city-district-street-road-door number-POI is realized.

[0083] 3. Clustering algorithm with enhanced spatial semantics: an improved DBSCAN algorithm is used to integrate text semantic similarity and spatial coordinate proximity to aggregate similar addresses. An outlier detection mechanism is introduced to analyze the free addresses again based on coordinate information, improving the robustness of clustering. Multi-scale hotspot analysis (such as regional, street, and POI levels) is supported to meet the fine-grained needs of different business scenarios.

[0084] The present application effectively solves the shortcomings of traditional methods in address noise processing, level completion, and semantic clustering, and provides more reliable technical support for geographic spatial intelligence analysis.

[0085] Address parsing accuracy improvement: through RAG-enhanced intelligent word segmentation and multi-level standardization, compared with traditional regular matching or keyword rules, the parsing accuracy of unstructured addresses can effectively solve the problems of wrong characters and missing levels.

[0086] Clustering effect optimization: combining text semantics and spatial coordinates, the address aggregation granularity is refined from the perspective of address text tagging, the DBSCAN clustering effect is optimized, and the analysis accuracy is improved.

[0087] Application level: suitable for intelligence analysis, hidden danger mining, and other fields to provide more accurate address matching and hotspot analysis capabilities.

[0088] Figure 6 The address information standardization flowchart provided by the present application is shown in Figure 6 In the fields of intelligence analysis and hidden danger mining, address data as a key spatial intelligence carrier faces the following main technical problems in standardization processing and hotspot mining:

[0089] Address text information processing: (1) Expression diversity problem: there are multiple forms of expression for the same address, such as Wuhan Traffic Management Bureau, Wuhan Traffic Management Bureau, and No. 2 of a certain road in a certain district; (2) Hierarchical information missing problem: the address of the alarm person is briefly expressed, which will affect the address positioning and comparison; (3) Address record misprints: due to the expression of the alarm person or the record deviation of the record person, there are homophonic words and misprints in the address, which are similar to the actual address description; (4) Wrong address: the recorded address is not the real address, or because of expression or record deviation, it is very different from the actual address.

[0090] Technical bottlenecks of hotspot mining: (1) Single dimension analysis: traditional methods only rely on text features (such as TF-IDF) or spatial coordinates (such as DBSCAN) information, and the effect cannot be guaranteed in the case of poor data quality. (2) Single analysis granularity and noise sensitivity: lack of multi-scale analysis capability, difficult to meet the analysis needs of regional level (such as administrative division) and point level (such as specific house number).

[0091] The application discloses an address standardization and hotspot mining algorithm based on RAG and spatial semantic calculation, which mainly includes three core technologies: (1) Address intelligent analysis based on RAG, high-precision address information extraction and word segmentation are realized by constructing a structured address knowledge base and using retrieval enhancement generation technology; (2) Multi-scale address standardization and error correction, stop word filtering, homophonic word conversion and other methods are used to complete multi-level address standardization processing from provincial level to POI level; (3) Clustering algorithm enhanced by spatial semantic, DBSCAN algorithm is improved and combined with text semantic and spatial coordinate information, combined with outlier detection mechanism to realize multi-scale hotspot analysis. The application effectively solves the expression diversity and hierarchical missing of address text, significantly improves the accuracy and efficiency of address standardization and hotspot mining, and can be widely applied in the fields of intelligence analysis and hidden danger mining.

[0092] Figure 7 The address information standardization processing based on RAG provided by the application is shown in the detailed flow chart as shown in Figure 7 The road topology network includes administrative division, street, road segment, intersection and POI. Multi-scale address analysis and standardization includes text description, large model address extraction, address word segmentation and part-of-speech tagging (road topology network realizes address word segmentation and part-of-speech tagging through RAG), address standardization and address semantic clustering completion. Address standardization includes stop word filtering, word deduplication, address number mapping, homophonic word replacement and synonymous word replacement. Intelligence mining analysis includes multi-scale spatial semantic calculation, DBSCAN clustering and coordinate-based effect optimization.

[0093] The address intelligent analysis process based on RAG is as follows:

[0094] The address information segment is coarsely extracted according to description information extraction by using a large model; an address standard information corpus is constructed by fusing an Internet road, a road library and historical high-quality labeled data; a standard address vector library is constructed relying on an administrative division, a road, a POI and the like address corpus; address segmentation and standardization processing are performed on the address information segment; and address entity labels are defined as follows: <province>, <city>, <district>, <street>, <road>, <poi>Information, using RAG technology, the extracted address information bge-m3 model is vectorized, and the vector similarity matching with the standard address library vector library is carried out, and the similar standard address auxiliary large model is efficiently searched to carry out address segmentation and address part-of-speech tagging. Each element involved in the address is stripped out and classified to form administrative division, street, road, intersection, POI standard police address segmentation information.

[0095] The multi-scale address standardization and error correction process is as follows:

[0096] The extracted address information is standardized by using stop word filtering, de-duplication, place number mapping, pinyin edit distance matching method, synonym replacement and other means. The specific method is: stop word filtering is to delete the information irrelevant to the address, such as "road curb, de, both sides"; de-duplication is to delete the redundant information in the address; place number mapping is to standardize the address collection of the case of ×× road × number, and to collect similar address descriptions into the same address; homophone replacement is to use pinyin and text similarity comparison to identify and replace synonyms, such as "certain district entertainment road" is misrecorded as "certain district entertainment road" due to telephone listening artificial error, then the same sound address is extracted from the standard address library using the same sound word, and the text distance matching algorithm is used to identify and replace the information with the same voice and similar text; synonym replacement is to use the synonym library to standardize the address recording of different descriptions of the same address, such as "traffic management bureau" and "certain district entertainment road 2".

[0097] The edit distance (text distance matching algorithm) is calculated as follows:

[0098] Suppose two address strings are X and Y, where the length of X is n and the length of Y is m. Define ED[i,…,j] as the minimum edit distance of X[1,…,i] to Y[1,…,j]. X[1,…,i] represents the first i characters of X, Y[1,…,j] represents the first j characters of Y, and ED[n,m] is the minimum edit distance of X and Y.

[0099] Then the minimum edit distance of X and Y is calculated as:

[0100]

[0101] Where, when x i ≠y j , Otherwise, 0.

[0102] The clustering algorithm process of spatial semantic enhancement is as follows:

[0103] The missing information part in the extracted police case address information fragment description is completed by using a semantic clustering algorithm. Specifically, the DBSCAN semantic clustering algorithm is used to aggregate the similar addresses after analysis, the address distribution rule is analyzed, and the administrative division and road information in part of the address information are corrected and completed, such as "traffic management bureau". According to historical information, the spatial semantic similarity is calculated, the address information is converged, the administrative division and road information are completed through comprehensive analysis, and the traffic management bureau of the public security bureau in a certain road in a certain district is processed.

[0104] The clustering algorithm steps are as follows:

[0105] (1) Parameter initialization ε and x.

[0106] ε neighborhood: the area within a radius of ε of a given data object p.

[0107] Core object: a data object containing at least x points (data objects) in the ε neighborhood.

[0108] (2) Data aggregation.

[0109] The ε neighborhood of each data point in the data set is traversed to find categories. If the number of data points in the ε neighborhood of p is not less than x, a category of core objects is constructed, and direct density reachable objects are repeatedly searched until no new objects are added, and the clustering ends.

[0110] Outlier detection and reanalysis mechanism: based on coordinate information, set the density threshold value δ, mark the core point / boundary point / noise point. Delete the detected outliers. Based on coordinate information, construct a kd-tree to redistribute the free points, and enhance the clustering effect. Figure 8 The clustering effect diagram provided in the present application.

[0111] Figure 9 The address information standardization processing device structure diagram based on RAG provided in the present application includes:

[0112] The first determination module 11 is configured to determine the first address information to be processed in the text description information based on a large model, determine the first address vector corresponding to the first address information based on the retrieval enhancement generation technology RAG, determine the first similarity between the first address vector and at least one standard address vector in the standard address library, determine at least one second address vector in the standard address library that matches the first address vector according to the first similarity, generate a prompt word based on the standard address information corresponding to the at least one second address vector in the standard address library, and determine the second address information in the text description information based on the prompt word and the large model.

[0113] The second determining module 12 is configured to determine semantic similarity between the second address information and pre-stored hierarchical complete address information, determine third address information matched with the second address information according to the semantic similarity, and perform hierarchical completion on the second address information according to the third address information to obtain target address information.

[0114] The first determining module 11 is specifically configured to determine a first sub-similarity between the first address vector and at least one road standard address vector in a road standard address library, select at least one road standard address vector in a descending order of the first sub-similarity as at least one second address vector matched with the first address vector in the road standard address library, determine a second sub-similarity between the first address vector and at least one street standard address vector in a street standard address library, select at least one street standard address vector in a descending order of the second sub-similarity as at least one second address vector matched with the first address vector in the street standard address library, determine a third sub-similarity between the first address vector and at least one POI standard address vector in a POI standard address library, and select at least one POI standard address vector in a descending order of the third sub-similarity as at least one second address vector matched with the first address vector in the POI standard address library.

[0115] The second determining module 12 is specifically configured to determine hierarchical complete address information corresponding to the maximum semantic similarity as third address information matched with the second address information.

[0116] The second determining module 12 is specifically configured to determine first address segmentation information of the second address information, determine a first hierarchical level to be completed in the second address information according to a hierarchical level corresponding to the first address segmentation information, determine second address segmentation information of the third address information, determine second address segmentation information corresponding to the first hierarchical level according to a hierarchical level corresponding to the second address segmentation information, and perform hierarchical completion on the second address information according to the second address segmentation information corresponding to the first hierarchical level to obtain target address information.

[0117] The first determining module 11 is further configured to determine at least one third address segmentation information of the second address information; the third address segmentation information comprises at least one of road address segmentation information, street address segmentation information and POI address segmentation information; for the at least one third address segmentation information, at least one standard address information homonymic to the third address segmentation information in a standard address library corresponding to the third address segmentation information is determined; a text similarity between the third address segmentation information and the at least one standard address information is determined by using a text distance matching algorithm; if a maximum text similarity is greater than a preset similarity threshold, a standard address information corresponding to the maximum text similarity is used to replace the third address segmentation information.

[0118] The first determining module 11 is further configured to match the second address information obtained after the third address segmentation information is replaced with pre-stored non-address information, determine and delete non-address information in the second address information; repeated address information in the second address information is recognized, and the repeated address information is de-duplicated; a road number in the second address information is mapped to a standard road number range by using a preset road number mapping rule; for the at least one third address segmentation information, if the third address segmentation information exists in a pre-stored synonym library, a synonymous standard address information corresponding to the third address segmentation information in the synonym library is used to replace the third address segmentation information.

[0119] The apparatus further comprises:

[0120] The clustering module 13 is configured to perform clustering processing on at least one address information in an address information library containing the target address information by using a DBSCAN semantic clustering algorithm, to obtain at least one cluster; and determine a hot address area according to a quantity of address information contained in the at least one cluster.

[0121] The clustering module 13 is further configured to, for at least one outlier point obtained after the clustering processing, determine other address information points in a preset area containing the outlier point, if the other address information points contain a non-outlier point, attribute the outlier point to a cluster in which the non-outlier point is located; and if the other address information points are all outlier points, cluster the outlier point and the other address information into one cluster.

[0122] The application further provides an electronic device, such as Figure 10 As shown in the figure, the electronic device comprises a processor 21, a communication interface 22, a memory 23 and a communication bus 24, wherein the processor 21, the communication interface 22 and the memory 23 complete mutual communication through the communication bus 24;

[0123] The memory 23 stores a computer program which, when executed by the processor 21, causes the processor 21 to perform any of the above method steps.

[0124] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For the sake of representation, only one thick line is shown in the figure, but it does not mean that there is only one bus or only one type of bus.

[0125] The communication interface 22 is used for communication between the above electronic device and other devices.

[0126] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0127] The processor can be a general-purpose processor, including a central processing unit, a network processor (NP), and the like; can also be a Digital Signal Processing (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and the like.

[0128] The present application also provides a computer storage readable storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device performs the above method steps.

[0129] The present application provides a computer program product, which includes an executable program which, when executed by a processor, implements the method.

[0130] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present application.

[0131] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.< / poi>

Claims

1. A method for standardizing address information based on RAG, characterized in that, The method includes: Based on a large model, the first address information to be processed in the text description information is determined; based on the Retrieval Enhancement Generation (RAG) technique, the first address vector corresponding to the first address information is determined; the first similarity between the first address vector and at least one standard address vector in the standard address library is determined; based on the first similarity, at least one second address vector in the standard address library that matches the first address vector is determined; prompt words are generated based on the standard address information corresponding to the at least one second address vector in the standard address library; based on the prompt words and the large model, the second address information in the text description information is determined. Determine the semantic similarity between the second address information and at least one pre-saved address information with complete hierarchical structure; determine the third address information that matches the second address information based on the semantic similarity; perform hierarchical completion on the second address information based on the third address information to obtain the target address information.

2. The method as described in claim 1, characterized in that, Determining a first similarity between the first address vector and at least one standard address vector in a standard address library, and determining at least one second address vector in the standard address library that matches the first address vector based on the first similarity, includes: Determine the first sub-similarity between the first address vector and at least one road standard address vector in the road standard address database, and select at least one road standard address vector in descending order of the first sub-similarity as at least one second address vector in the road standard address database that matches the first address vector; Determine the second sub-similarity between the first address vector and at least one street standard address vector in the street standard address database, and select at least one street standard address vector in descending order of the second sub-similarity as at least one second address vector in the street standard address database that matches the first address vector; Determine the third sub-similarity between the first address vector and at least one POI standard address vector in the POI standard address library, and select at least one POI standard address vector in descending order of the third sub-similarity as at least one second address vector in the POI standard address library that matches the first address vector.

3. The method as described in claim 1, characterized in that, The third address information that matches the second address information is determined based on the semantic similarity, including: The complete address information corresponding to the highest semantic similarity is used as the third address information to match the second address information.

4. The method as described in claim 1, characterized in that, The target address information is obtained by performing hierarchical completion on the second address information based on the third address information, including: Determine the first address segmentation information of the second address information, and determine the first level to be completed in the second address information based on the level corresponding to the first address segmentation information; The second address segmentation information of the third address information is determined. Based on the level corresponding to the second address segmentation information, the second address segmentation information corresponding to the first level is determined. Based on the second address segmentation information corresponding to the first level, the second address information is completed hierarchically to obtain the target address information.

5. The method as described in claim 1, characterized in that, After determining the second address information in the text description information based on the prompt words and the large model, and before determining the semantic similarity between the second address information and at least one pre-saved address information with complete hierarchical structure, the method further includes: Determine at least one third address segmentation information of the second address information; wherein, the third address segmentation information includes at least one of road address segmentation information, street address segmentation information, and POI address segmentation information; For the at least one third address segmentation information, determine at least one standard address information in the standard address library that is homophonous with the third address segmentation information; determine the text similarity between the third address segmentation information and the at least one standard address information through a text distance matching algorithm; if the maximum text similarity is greater than a preset similarity threshold, replace the third address segmentation information with the standard address information corresponding to the maximum text similarity.

6. The method as described in claim 5, characterized in that, After replacing the third address segmentation information with the standard address information corresponding to the highest text similarity, and before determining the semantic similarity between the second address information and at least one pre-saved hierarchical complete address information, the method further includes: The second address information obtained after replacing the third address segmentation information is matched with the pre-saved non-address information to determine and delete the non-address information in the second address information; Identify duplicate address information in the second address information and perform deduplication on the duplicate address information; By using preset road number mapping rules, the road number in the second address information is mapped to a standard road number range; For the at least one third address segmentation information, if the third address segmentation information exists in the pre-saved thesaurus, the third address segmentation information is replaced with the standard thesaurus address information corresponding to the third address segmentation information in the thesaurus.

7. The method as described in claim 1, characterized in that, The method further includes: The DBSCAN semantic clustering algorithm is used to cluster at least one address information in the address information database, including the target address information, to obtain at least one cluster; based on the number of address information contained in the at least one cluster, hotspot address regions are determined.

8. The method as described in claim 7, characterized in that, After obtaining at least one cluster, before determining the hotspot address region based on the number of address information contained in the at least one cluster, the method further includes: For at least one outlier obtained after clustering, other address information points within a preset area, including the outlier, are identified. If the other address information points include non-outliers, the outlier is assigned to the cluster containing the non-outliers. If all other address information points are outliers, the outlier and the other address information points are clustered into one cluster.

9. A RAG-based address information standardization processing device, characterized in that, The device includes: The first determining module is used to determine the first address information to be processed in the text description information based on a large model; determine the first address vector corresponding to the first address information based on the retrieval enhancement generation technology (RAG); determine the first similarity between the first address vector and at least one standard address vector in the standard address library; determine at least one second address vector in the standard address library that matches the first address vector based on the first similarity; generate prompt words based on the standard address information corresponding to the at least one second address vector in the standard address library; and determine the second address information in the text description information based on the prompt words and the large model. The second determining module is used to determine the semantic similarity between the second address information and at least one pre-saved address information with complete hierarchical structure, determine the third address information that matches the second address information based on the semantic similarity, and perform hierarchical completion on the second address information based on the third address information to obtain the target address information.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-8.

Images