Address information processing method and device, electronic equipment and storage medium
By segmenting the address information database and processing it with cosine similarity, the problem of address query accuracy caused by voice input errors was solved, achieving efficient address error correction and query optimization.
Patent Information
- Application Number
- CN202411168153.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
AI Technical Summary
In existing technologies, address queries using voice input suffer from low accuracy due to recognition errors and accent differences. Deep learning models are limited in application on mobile devices and consume a lot of computing resources, which affects user experience.
By segmenting the address information database to generate multiple substrings, and using the trained representation vectors to calculate the model and the generation model, preliminary screening and error correction are performed based on cosine similarity, thereby reducing the amount of data and computational complexity of the deep learning model.
It can rapidly narrow down the target range in large-scale data, reduce computational complexity, improve processing efficiency and user experience, and enhance the model's ability to correct speech input errors.
Smart Images

Figure CN119025607A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of information retrieval and natural language processing, and in particular to an address information processing method, device, electronic device and storage medium. Background Art
[0002] In daily life, people often need to search for specific POIs (Point of Interest), such as restaurants, banks, supermarkets, etc. With the popularization of mobile Internet and smart devices, more and more users are searching through voice input. However, voice input often affects the accuracy of search results due to factors such as recognition errors and accent differences. For example, if a user wants to search for "xx bank", the result of voice recognition is "xx bank".
[0003] In related technologies, although deep learning models can theoretically provide more accurate error correction matching, their demand for large amounts of training data and consumption of computing resources limit their application on mobile devices. At the same time, the long inference time of deep learning models also makes it difficult to implement the model, reducing the user experience. Summary of the invention
[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.
[0005] To this end, the first purpose of the present application is to propose an address information processing method to achieve a degree of similarity between multiple first substrings obtained by cutting at least one first address information in an address information database and the second address information input by the user, to perform preliminary screening of multiple first substrings, to reduce the amount of data that needs to be processed by a deep learning model, to reduce computational complexity, and to improve processing efficiency and user experience.
[0006] The second objective of the present application is to provide an address information processing device.
[0007] The third objective of the present application is to provide an electronic device.
[0008] A fourth objective of the present application is to provide a computer-readable storage medium.
[0009] A fifth object of the present application is to provide a computer program product.
[0010] To achieve the above-mentioned purpose, the first aspect embodiment of the present application proposes an address information processing method, including: in response to a user input of second address information to be queried, reading first address information in an address information database, wherein the first address information is generated based on a pre-set standard address description; determining multiple second substrings similar to the second address information from the multiple first substrings according to the similarity between the multiple first substrings obtained by cutting the first address information and the second address information; using a trained representation vector calculation model to generate a representation vector for the second address information and a representation vector for each of the second substrings; and determining whether to use the trained generation model to perform correction processing on the second address information according to the cosine similarity between the representation vector of the second address information and the representation vector of each of the second substrings in the feature space.
[0011] To achieve the above-mentioned purpose, the second aspect of the present application proposes an address information processing device, including: a reading module, used to read the first address information in the address information database in response to the user input of the second address information to be queried, wherein the first address information is generated based on a pre-set standard address description; a first determination module, used to determine a plurality of second substrings similar to the second address information from the plurality of first substrings according to the similarity between the plurality of first substrings obtained by cutting the first address information and the second address information; a first generation module, used to use a trained representation vector calculation model to generate a representation vector for the second address information and a representation vector for each of the second substrings; a second determination module, used to determine whether to use the trained generation model to perform correction processing on the second address information according to the cosine similarity between the representation vector of the second address information and the representation vector of each of the second substrings in the feature space.
[0012] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement the address information processing method as described in the above embodiment of the present application.
[0013] To achieve the above objectives, the fourth aspect of the present application proposes a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of an electronic device, the electronic device can execute the address information processing method described in the above embodiment of the present application.
[0014] To achieve the above objectives, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the address information processing method as described in the above embodiments of the present application.
[0015] The address information processing method, device, electronic device and storage medium provided by the present application read the first address information in the address information database in response to the second address information to be queried input by the user, wherein the first address information is generated based on a preset standard address description; according to the similarity between the multiple first substrings obtained by cutting the first address information and the second address information, multiple second substrings similar to the second address information are determined from the multiple first substrings; a representation vector of the second address information and the representation vectors of each second substring are generated by using a trained representation vector calculation model; according to the cosine similarity between the representation vector of the second address information and the representation vector of each second substring in the feature space, it is determined whether to use the trained generation model to correct the second address information. Thus, by preliminarily screening the multiple first substrings based on the similarity between the multiple first substrings obtained by cutting each first address information in the address information database and the second address information input by the user, the target range can be quickly narrowed down in large-scale data, the amount of data that needs to be processed by the deep learning model can be reduced, the computational complexity can be reduced, and the processing efficiency and user experience can be improved.
[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0018] Figure 1 It is a flowchart of an address information processing method provided in an embodiment of the present application.
[0019] Figure 2 It is a flowchart of another address information processing method provided in an embodiment of the present application.
[0020] Figure 3 It is a flowchart of another address information processing method provided in an embodiment of the present application.
[0021] Figure 4 It is a schematic diagram of the principle of an address information processing method provided in an embodiment of the present application.
[0022] Figure 5 It is a structural diagram of an address information processing device provided in an embodiment of the present application.
[0023] Figure 6 It is a schematic diagram of the structure of an electronic device 600 shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0024] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0025] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.
[0026] Among them, it should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of relevant laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.
[0027] Related technologies, methods for handling speech input errors mainly include string matching based on edit distance, keyword-based search algorithms, and deep learning models. However, these methods face the following challenges when processing large-scale POI data:
[0028] Although string matching methods based on edit distance can handle simple spelling errors, they are limited in effectiveness when faced with large deviations caused by speech recognition errors;
[0029] Keyword-based search algorithms, such as the BM25 algorithm, rely on text matching of keywords and lack effective processing capabilities for fuzzy errors in speech recognition.
[0030] Although deep learning models can theoretically provide more accurate error correction matching, their demand for large amounts of training data and consumption of computing resources limit their application on mobile devices. At the same time, the long inference time of deep learning models also makes model implementation difficult and reduces user experience.
[0031] In response to at least one of the above problems, the present application proposes an address information processing method, device, electronic device and storage medium.
[0032] The following describes the address information processing method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.
[0033] Figure 1It is a flowchart of an address information processing method provided in an embodiment of the present application.
[0034] It should be noted that the address information processing method provided in the embodiment of the present application can be executed by an address information processing device, wherein the address information processing device can be implemented by software and / or hardware. The address information processing device can be an electronic device, or can be configured in an electronic device to implement the address information processing function. The embodiment of the present application is described by taking the address information processing method configured in an electronic device as an example.
[0035] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.
[0036] like Figure 1 As shown, the address information processing method may include the following:
[0037] Step 101: In response to a user inputting second address information to be queried, first address information in an address information database is read.
[0038] The first address information is generated based on a preset standard address description.
[0039] The address information database may be any database storing at least one first address information.
[0040] The first address information and the second address information are only used to distinguish that one is the address information in the address information database and the other is the address information to be queried input by the user.
[0041] Step 102: Determine a plurality of second substrings close to the second address information from the plurality of first substrings according to the degree of similarity between the plurality of first substrings obtained by cutting the first address information and the second address information.
[0042] Among them, multiple first substrings can be obtained after cutting any first address information in the address information database.
[0043] Among them, the number of second substrings close to the second address information determined from multiple first substrings can be pre-set. It should be noted that there is no restriction on the number of pre-set second substrings in the embodiment of the present application. For example, the number of second substrings determined can be pre-set to 10, 16, etc.
[0044] In order to be able to quickly narrow down the target range in large-scale data, reduce the amount of data that needs to be processed by the deep learning model, and reduce the computational complexity, in an embodiment of the present application, at least one first address information in the address information database is cut in advance to obtain multiple first substrings, and then, based on the degree of similarity between each first substring and the second address information input by the user, multiple second substrings similar to the second address information are determined from the multiple first substrings obtained by cutting.
[0045] As a possible implementation method, a string processing algorithm can be used in combination with a word segmentation method in NLP (Natural Language Processing) technology to ensure that the multiple first substrings obtained after cutting any first address information in the address information database can cover all possible word segmentation substrings. For example, when cutting "China Post Savings Bank", all logically possible independent name fragments such as "China", "China Post", "Postal", "Savings", "Postal Savings", "Savings Bank", and "Postal Savings Bank" should be obtained.
[0046] Step 103: Use the trained representation vector calculation model to generate a representation vector for the second address information and a representation vector for each second substring.
[0047] Among them, the representation vector calculation model is a deep learning model, which can learn complex text features and contextual relationships, and provide a more accurate match between the second address information input by the user and each second substring.
[0048] In order to more accurately identify whether the second address information input by the user is incorrect, in an embodiment of the present application, a trained representation vector calculation model is used to generate a representation vector for the second address information and a representation vector for each second substring, so as to calculate the cosine similarity between the representation vector of the second address information and the representation vector of each second substring in the feature space, so as to identify whether the second address information input by the user is incorrect based on the cosine similarity between the representation vector of the second address information and the representation vector of each second substring in the feature space.
[0049] As a possible implementation manner, the second address information and each second substring may be input into a trained representation vector calculation model to obtain a representation vector of the second address information and a representation vector of each second substring output by the model.
[0050] Step 104 , determining whether to use the trained generation model to perform correction processing on the second address information based on the cosine similarity between the characterization vector of the second address information and the characterization vectors of each second substring in the feature space.
[0051] Among them, the generation model is a deep learning model that can learn complex text features and contextual relationships and restore incorrect address information to correct address information.
[0052] In order to improve the accuracy of error correction, in an embodiment of the present application, after using a trained representation vector calculation model to generate a representation vector for the second address information and a representation vector for each second substring, it is possible to identify whether the second address information input by the user is incorrect based on the cosine similarity between the representation vector of the second address information and the representation vector of each second substring in the feature space. In this way, if the second address information is incorrect, the trained generation model is used to correct the second address information and provide the user with the correct address information after the correction.
[0053] As a possible implementation method, based on the cosine similarity between the second address information and each second substring, a third substring with the highest cosine similarity is determined from each second substring; when the cosine similarity between the second address information and the third substring does not reach a preset threshold, the second address information is corrected using a trained generation model; when the cosine similarity between the second address information and the third substring reaches a preset threshold, the first address information corresponding to the third substring is provided to the user.
[0054] That is to say, the highest cosine similarity among the cosine similarities between the representation vector of the second address information and the representation vector of each second substring in the feature space can be compared with a preset threshold. If the highest cosine similarity is not greater than the preset threshold, it indicates that the second address information queried by the user is not in the address information database, that is, the second address information input by the user is incorrect. At this time, the trained generation model can be used to correct the second address information and provide the user with possible correct address information.
[0055] Among them, the preset threshold value can be any value, and there is no limitation on this in the embodiment of the present application. For example, the preset threshold value can be 0.85, 0.9, etc.
[0056] Among them, the first address information corresponding to the third substring may be one or more, and there is no limitation on this in the embodiment of the present application. For example, the third substring is "AA", and the first address information corresponding to the third substring may be at least one address information including "AA".
[0057] The address information processing method of the embodiment of the present application, in response to the user inputting the second address information to be queried, reads the first address information in the address information database, wherein the first address information is generated based on a pre-set standard address description; according to the similarity between the multiple first substrings obtained by cutting the first address information and the second address information, determines the second substring close to the second address information from the multiple first substrings; uses the trained representation vector calculation model to generate the representation vector of the second address information and the representation vector of the second substring; according to the cosine similarity between the representation vector of the second address information and the representation vector of the second substring in the feature space, determines whether to use the trained generation model to correct the second address information. Thus, by preliminarily screening the multiple first substrings based on the similarity between the multiple first substrings obtained by cutting each first address information in the address information database and the second address information input by the user, it is possible to quickly narrow down the target range in large-scale data, reduce the amount of data that needs to be processed by the deep learning model, reduce the computational complexity, and improve the processing efficiency and user experience.
[0058] In order to clearly explain how the present application determines multiple second substrings similar to the second address information from multiple first substrings obtained by cutting at least one first address information in the address information database according to the similarity between the second address information input by the user, the embodiment of the present application provides another address information processing method, Figure 2 It is a flowchart of another address information processing method provided in an embodiment of the present application.
[0059] like Figure 2 As shown, the address information processing method may include the following steps:
[0060] Step 201: In response to a user inputting second address information to be queried, first address information in an address information database is read.
[0061] Step 202: construct an inverted index for the first language characters included in the plurality of first substrings and the first symbol character corresponding to each first language character and representing the pronunciation of the first language character, so as to obtain an inverted index corresponding to each first language character and an inverted index corresponding to each first symbol character.
[0062] Among them, the inverted index corresponding to each first language character records multiple substring identifiers of the fourth substring in the first substring that contains the first language character and the frequency of occurrence of the first language character in the fourth substring, and the inverted index corresponding to each first symbol character records multiple substring identifiers of the fifth substring in the first substring that contains the first symbol character and the frequency of occurrence of the first symbol character in the fifth substring.
[0063] Among them, the first language characters can be characters of any language, such as Chinese single characters, English words, and so on.
[0064] Among them, the first symbol characters are used to represent the pronunciation of the corresponding first language characters. For example, when the first language characters are Chinese single characters, the first symbol characters can be single pinyin characters. It should be noted that the single pinyin characters here refer to the complete pinyin of a Chinese single character. For example, assuming that the first language character is "中", the first symbol character corresponding to this first language character is "zhong".
[0065] Among them, the fourth substring and the fifth substring are only used to distinguish one as the substring containing the first language characters among multiple first substrings, and the other as the substring containing the first symbol characters among multiple first substrings.
[0066] As a possible implementation, the first language characters and the first symbol characters corresponding to the first language characters included in multiple first substrings can be determined in the following way:
[0067] Perform word segmentation on any first substring to obtain at least one first language character included in the first substring; perform conversion processing on each first language character to obtain the first symbol character corresponding to each first language character, which represents the pronunciation of the first language character.
[0068] As a possible implementation, for any first substring among multiple first substrings, determine the occurrence frequency of the first language characters included in the first substring in the first substring, and the occurrence frequency of the first symbol characters corresponding to the first language characters included in the first substring in the first substring; construct an inverted index corresponding to each first language character according to the occurrence frequency of the first language characters included in each first substring in each first substring and the substring identifier of each first substring; construct an inverted index corresponding to each first symbol character according to the occurrence frequency of the first symbol characters corresponding to the first language characters included in each first substring in each first substring and the substring identifier of each first substring.
[0069] For example, assume that multiple first substrings are: 1. China; 2. China Post; 3. Postal Savings. Here, 1 is the substring identifier of the first substring "China", 2 is the substring identifier of the first substring "China Post", and 3 is the substring identifier of the first substring "Postal Savings". Among them, the first language characters included in the first substring "China" are "中" and "国", and the first symbol characters corresponding to the first language characters included in the first substring "China" are "zhong" and "guo"; the first language characters included in the first substring "China Post" are "中", "国", "邮", and "政", and the first symbol characters corresponding to the first language characters included in the first substring "China Post" are "zhong", "guo", "you", and "zheng"; the first language characters included in the first substring "Postal Savings" are "邮", "政", "储", and "蓄", and the first symbol characters corresponding to the first language characters included in the first substring "Postal Savings" are "you", "zheng", "chu", and "xu". Then, for any first substring, the occurrence frequency of the first language characters included in the first substring in the first substring, and the occurrence frequency of the first symbol characters corresponding to the first language characters included in the first substring in the first substring can be determined. The determination results are as follows:
[0070] The first substring "China":
[0071] First language character: Occurrence frequency of the first language character in the first substring
[0072] "中": 1, "国": 1
[0073] First symbol character corresponding to the first language character: Occurrence frequency of the first symbol character corresponding to the first language character in the first substring
[0074] "zhong": 1, "guo": 1
[0075] The first substring "China Post":
[0076] First language character: Occurrence frequency of the first language character in the first substring
[0077] "中": 1, "国": 1, "邮": 1, "政": 1
[0078] First symbol character corresponding to the first language character: Occurrence frequency of the first symbol character corresponding to the first language character in the first substring
[0079] "zhong": 1, "guo": 1, "you": 1, "zheng": 1
[0080] The first substring "Postal Savings"
[0081] First language characters: The occurrence frequency of first language characters in the first substring
[0082] "You": 1, "Zheng": 1, "Chu": 1, "Xu": 1
[0083] First symbol characters corresponding to the first language characters: The occurrence frequency of the first symbol characters corresponding to the first language characters in the first substring
[0084] "you": 1, "zheng": 1, "chu": 1, "xu": 1
[0085] Furthermore, the following inverted index can be constructed for the first language symbols ("Zhong", "Guo", "You", "Zheng", "Chu", "Xu") contained in the above first substring:
[0086] First language characters: (Substring identifier of the fourth substring: The occurrence frequency of the first language characters in the fourth substring)
[0087] "Zhong": (1: 1), (2: 1)
[0088] "Guo": (1: 1), (2: 1)
[0089] "You": (2: 1), (3: 1)
[0090] "Zheng": (2: 1), (3: 1)
[0091] "Chu": (3: 1)
[0092] "Xu": (3: 1)
[0093] And the following inverted index can be constructed for the first symbol characters ("zhong", "guo", "you", "zheng", "chu", "xu") corresponding to the first language symbols ("Zhong", "Guo", "You", "Zheng", "Chu", "Xu") contained in the above first substring:
[0094] First symbol characters corresponding to the first language characters: (Substring identifier of the fifth substring: The occurrence frequency of the first symbol characters corresponding to the first language characters in the fifth substring)
[0095] "zhong": (1: 1), (2: 1)
[0096] "guo": (1: 1), (2: 1)
[0097] "you": (2: 1), (3: 1)
[0098] "zheng": (2: 1), (3: 1)
[0099] "chu": (3: 1)
[0100] "xu": (3:1)
[0101] It should be noted that in order to improve the accuracy of error correction, in an embodiment of the present application, when the first language character is a Chinese word and the first symbol character is a Pinyin word, for each Pinyin word, a fuzzy Pinyin of the Pinyin can be added. For example, for zh, ch, sh, z, c, s are added. That is, for the above-mentioned multiple first substrings of 1. China; 2. China Post; 3. Postal Savings, when constructing the inverted index corresponding to the first symbol character, in addition to constructing the inverted indexes corresponding to the six first symbol characters "zhong", "guo", "you", "zheng", "chu", and "xu", it is also necessary to construct the inverted indexes corresponding to the three Pinyin words "zong", "zeng", and "cu".
[0102] Step 203, based on the inverted index corresponding to each first language character, the inverted index corresponding to each first symbol character, the second language character included in the second address information, and the second symbol character corresponding to each second language character representing the pronunciation of the second language character, determine the term frequency-inverse document frequency TF-IDF score corresponding to at least one target substring of the multiple first substrings that contains the second language character and / or the second symbol character.
[0103] As a possible implementation manner, for any second language character included in the second address information, determine the frequency of occurrence of the second language character in the second address information and the frequency of occurrence of the second symbol character corresponding to the second language character in the second address information; when any first language character is the second language character included in the second address information, determine at least one fourth substring recorded in the inverted index corresponding to the first language character as a target substring; when any first symbol character is the second symbol character corresponding to the second language character included in the second address information, determine at least one fifth substring recorded in the inverted index corresponding to the first symbol character as a target substring; and determine the TF-IDF score corresponding to each target substring based on the frequency of occurrence of the second language character in each target substring, the frequency of occurrence of the second symbol character corresponding to the second language character in each target substring, the frequency of occurrence of the second language character in the second address information, and the frequency of occurrence of the second symbol character corresponding to the second language character in the second address information.
[0104] As a possible implementation, when determining the TF-IDF scores corresponding to each target substring, the TF-IDF scores corresponding to each target substring can be initialized to a preset value first; then, it is successively determined whether the occurrence frequency of the second-language characters in each target substring is the same as the occurrence frequency of the second-language characters in the second address information, so that when they are the same, the TF-IDF score corresponding to the target substring is increased by a first set value; and, it is successively determined whether the occurrence frequency of the second symbol characters corresponding to the second-language characters in each target substring is the same as the occurrence frequency of the second symbol characters corresponding to the second-language characters in the second address information, so that when they are the same, the TF-IDF score corresponding to the target substring is increased by a second set value. Thus, after the determination is completed, the TF-IDF scores corresponding to each target substring can be obtained.
[0105] Among them, the preset value, the first set value, and the second set value can be any numerical values, and there is no limitation in this regard in the embodiments of the present application. For example, the preset value can be 0, the first set value and the second set value can be 1, or the preset value can be 1, the first set value can be 2, the second set value can be 3, and so on.
[0106] It should be noted that as shown in the above examples, the first set value and the second set value can be the same value. For example, both the first set value and the second set value are 1, or they can be different values. For example, the first set value is 1 and the second set value is 2, and so on. There is no limitation in this regard in the embodiments of the present application.
[0107] For example, for the above examples where the multiple first substrings are 1. China; 2. China Post; 3. Postal Savings, assuming that the second address information input by the user is "Postal", where the second-language characters included in the second address information "Postal" are "you" and "zheng", and the second symbol characters corresponding to the second-language characters in the second address information "Postal" are "you" and "zheng", then for any second-language character included in the second address information, the occurrence frequency of the second-language character in the second address information and the occurrence frequency of the second symbol character corresponding to the second-language character in the second address information can be determined. The determination results are as follows:
[0108] Second address information "Postal":
[0109] Second-language character: Occurrence frequency of the second-language character in the second address information
[0110] "you": 1, "zheng": 1
[0111] Second symbol character corresponding to the second-language character: Occurrence frequency of the second symbol character corresponding to the second-language character in the second address information
[0112] "you": 1, "zheng": 1
[0113] Furthermore, in a case where any first language character is a second language character included in the second address information, at least one fourth substring recorded in the inverted index corresponding to the first language character can be determined as a target substring. That is, at least one fourth substring (the first substring "China Post" with substring identifier 2 and the first substring "Post Savings" with substring identifier 3) recorded in the inverted index ("you": (2: 1), (3: 1)) corresponding to the first language character "you" is determined as the target substring, and at least one fourth substring (the first substring "China Post" with substring identifier 2 and the first substring "Post Savings" with substring identifier 3) recorded in the inverted index ("zheng": (2: 1), (3: 1)) corresponding to the first language character "zheng" is determined as the target substring.
[0114] And in a case where any first symbol character is a second symbol character corresponding to a second language character included in the second address information, at least one fifth substring recorded in the inverted index corresponding to the first symbol character can be determined as a target substring. That is, at least one fifth substring (the first substring "China Post" with substring identifier 2 and the first substring "Post Savings" with substring identifier 3) recorded in the inverted index ("you": (2: 1), (3: 1)) corresponding to the first symbol character "you" is determined as the second target substring, and at least one fifth substring (the first substring "China Post" with substring identifier 2 and the first substring "Post Savings" with substring identifier 3) recorded in the inverted index ("zheng": (2: 1), (3: 1)) corresponding to the first symbol character "zheng" is determined as the target substring.
[0115] In summary, the target substrings are the first substring "China Post" with substring identifier 2 and the first substring "Postal Savings" with substring identifier 3. Furthermore, based on the occurrence frequency of the second language characters in each target substring, the occurrence frequency of the second symbol characters corresponding to the second language characters in each target substring, the occurrence frequency of the second language characters in the second address information, and the occurrence frequency of the second symbol characters corresponding to the second language characters in the second address information, the TF-IDF scores corresponding to each target substring can be determined. For example, first initialize the TF-IDF scores corresponding to the first substring "China Post" with substring identifier 2 and the first substring "Postal Savings" with substring identifier 3 to 0. Then, for the second language character "邮" in the second address information "邮政", since the occurrence frequency of "邮" in the first substring "China Post" with substring identifier 2 is the same as the occurrence frequency of "邮" in the second address information "邮政", the TF-IDF score corresponding to the first substring "China Post" with substring identifier 2 is incremented by 1 point and is currently 1 point. Similarly, the TF-IDF score corresponding to the first substring "Postal Savings" with substring identifier 3 is also incremented by 1 point and is currently 1 point. For the second language character "政" in the second address information "邮政", the TF-IDF scores corresponding to the first substring "China Post" with substring identifier 2 and the first substring "Postal Savings" with substring identifier 3 are each incremented by 1 point and are currently both 2 points. For the second symbol character "you" corresponding to the second language character "邮" in the second address information "邮政", the TF-IDF scores corresponding to the first substring "China Post" with substring identifier 2 and the first substring "Postal Savings" with substring identifier 3 are each incremented by 1 point and are currently both 3 points. For the second symbol character "zheng" corresponding to the second language character "政" in the second address information "邮政", the TF-IDF scores corresponding to the first substring "China Post" with substring identifier 2 and the first substring "Postal Savings" with substring identifier 3 are each incremented by 1 point and are currently both 4 points. The judgment ends, and the TF-IDF scores corresponding to the first substring "China Post" with substring identifier 2 and the first substring "Postal Savings" with substring identifier 3 are both 4 points.
[0116] Step 204: Sort each target substring according to the TF-IDF score corresponding to each target substring, so as to determine the second substring based on the sorted target substrings.
[0117] As a possible implementation method, each target substring can be sorted according to a set sorting order based on the TF-IDF score corresponding to each target substring to obtain each sorted target substring; when the sorting order is set to ascending sorting, the target substring with the last set number of digits in each sorted target substring is determined as the second substring; when the sorting order is set to descending sorting, the target substring with the first set number of digits in each sorted target substring is determined as the second substring.
[0118] The number of set bits can be any value, which is not limited in the embodiments of the present application. For example, the number of set bits can be 2 bits, 5 bits, etc.
[0119] It should be noted that when the number of target substrings is less than the set number of bits, all target substrings can be directly determined as the second substring.
[0120] Step 205: Use the trained representation vector calculation model to generate a representation vector for the second address information and a representation vector for each second substring.
[0121] Step 206 , determining whether to use the trained generation model to perform correction processing on the second address information based on the cosine similarity between the characterization vector of the second address information and the characterization vectors of each second substring in the feature space.
[0122] It should be noted that the execution process of step 201 and steps 204 to 205 can be implemented in any way in the embodiments of the present application, and the embodiments of the present application do not limit this and will not be repeated.
[0123] The address information processing method of the embodiment of the present application builds an inverted index for the first language characters contained in the multiple first substrings and the first symbol characters corresponding to each first language character representing the pronunciation of the first language character, so as to obtain the inverted index corresponding to each first language character and the inverted index corresponding to each first symbol character; according to the inverted index corresponding to each first language character, the inverted index corresponding to each first symbol character, the second language character contained in the second address information, and the second symbol characters corresponding to each second language character representing the pronunciation of the second language character, the term frequency-inverse document frequency TF-IDF score corresponding to at least one target substring containing the second language character and / or the second symbol character in the multiple first substrings is determined; according to the TF-IDF score corresponding to each target substring, each target substring is sorted, so as to determine the second substring according to each sorted target substring. Thus, by using TF-IDF for preliminary screening and quickly recalling samples that may be related to the user query, the target range can be quickly narrowed in large-scale data and the initial efficiency of retrieval can be improved.
[0124] In order to clearly illustrate the training process of the representation vector calculation model and the generation model in the present application, the embodiment of the present application provides another address information processing method. Figure 3 It is a flowchart of another address information processing method provided in an embodiment of the present application.
[0125] like Figure 3 As shown, the address information processing method may include the following steps:
[0126] Step 301: for any first address information in the address information database, the first address information is segmented to obtain a plurality of first training substrings corresponding to the first address information.
[0127] The execution process of this step may specifically refer to step 102 in the above embodiment, and the principle is the same, which will not be repeated here.
[0128] Step 302: determine a candidate training substring from the plurality of first training substrings, and determine a plurality of second training substrings similar to the candidate training substring from the target training substring according to the similarity between the target training substring and the candidate training substring.
[0129] The target training substring is each first training substring among the plurality of first training substrings except the candidate training substring.
[0130] In an embodiment of the present application, a first training substring can be selected from multiple first training substrings corresponding to any first address information as a candidate training substring, and based on the degree of similarity between each first training substring other than the candidate training substring in the multiple first training substrings corresponding to the first address information and the candidate training substring, multiple second training substrings similar to the candidate training substring are determined from the target training substring.
[0131] As a possible implementation, an inverted index may be constructed for the third language characters included in the target training substring and the third symbol characters corresponding to each third language character and indicating the pronunciation of the third language character, so as to obtain an inverted index corresponding to each third language character and an inverted index corresponding to each third symbol character, wherein the inverted index corresponding to each third language character records the substring identifier of the first target training substring containing the third language character and the occurrence frequency of the third language character in the first target training substring, and the inverted index corresponding to each third symbol character records the substring identifier of the second target training substring containing the third symbol character in the target training substring. The method comprises the steps of: determining a TF-IDF score corresponding to at least one third target training substring containing the fourth language character and / or the fourth symbol character in the target training substring according to the inverted index corresponding to each third language character, the inverted index corresponding to each third symbol character, the fourth language character contained in the candidate training substring, and the fourth symbol character representing the pronunciation of the fourth language character corresponding to each fourth language character; and sorting the third target training substrings according to the TF-IDF scores corresponding to the third target training substrings to determine the second training substring according to the sorted third target training substrings.
[0132] The execution process of determining the second training substring may specifically refer to steps 201 to 203 in the previous embodiment, and the principles are the same, which will not be described in detail herein.
[0133] Therefore, by using TF-IDF for initial screening, the amount of data that needs to be processed by the deep learning model is reduced, thereby reducing the computational complexity and increasing the processing speed, which can significantly improve the user experience.
[0134] Step 303: Generate a training batch corresponding to the first address information according to the candidate training substrings and the plurality of second training substrings.
[0135] As a possible implementation method, for the candidate training substring and the multiple second training substrings, a preset error generation rule can be used to generate adversarial samples corresponding to the candidate training substring and adversarial samples corresponding to each second training substring; the candidate training substring and the multiple second training substrings, as well as the adversarial samples corresponding to the candidate training substring and the adversarial samples corresponding to each second training substring, are determined as a training batch corresponding to the first address information. Thus, by generating adversarial samples and adding these adversarial samples to the training batch, the model can learn more real-world changes and noise, and enhance the model's generalization and error correction capabilities for unseen error samples.
[0136] The preset error generation rules may include the following rules: replacing characters (for example, when the substring is a Chinese substring, any Chinese character may be replaced with a homophone character), randomly inserting or deleting characters, and the like.
[0137] As an example, errors that may occur in user input, such as typos, homophone errors, etc., may be simulated for the candidate training substring and each second training substring.
[0138] Step 304, according to each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample, simultaneously train the representation vector calculation model and the generation model to obtain a trained representation vector calculation model and a trained generation model.
[0139] As a possible implementation method, each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample can be input into a representation vector calculation model, so that the cosine similarity in the feature space between the representation vector of any training sample generated by the trained representation vector calculation model and the representation vector of the adversarial sample corresponding to the training sample is higher than the cosine similarity in the feature space between the representation vector of the training sample and the representation vector of the adversarial sample corresponding to any training sample other than the training sample, wherein the representation vector calculation model uses a preset loss function to identify the correspondence between each training sample in any training batch and the adversarial sample corresponding to each training sample; for any training batch corresponding to each first address information, the adversarial sample corresponding to each training sample in the training batch is used as the input of the generation model, and each training sample in the training batch is used as the output of the generation model, and the generation model is trained so that the trained generation model can restore the adversarial sample corresponding to any training sample to the training sample. Optionally, the preset loss function can be a cross entropy loss function.
[0140] Therefore, compared with the related art in which only the cosine similarity of the representation vectors of a group of related samples is calculated in the feature space, in the present application, considering that speech input often has more uncertainties and errors (such as accent differences, unclear pronunciation, etc.), by calculating the cosine similarity of the representation vector of any training sample and the representation vector of the adversarial sample corresponding to all training samples (including the training sample) in the training batch of the training sample during the training of the representation vector calculation model, the model's processing of speech input is optimized, and it is also beneficial to enhance the accuracy of the generative model in restoring the adversarial sample corresponding to any training sample to the training sample.
[0141] The address information processing method of the embodiment of the present application is to cut the first address information for any first address information in the address information database to obtain multiple first training substrings corresponding to the first address information; determine the candidate training substring from the multiple first training substrings, and determine multiple second training substrings similar to the candidate training substring from the target training substring according to the similarity between the target training substring and the candidate training substring; generate a training batch corresponding to the first address information according to the candidate training substring and the multiple second training substrings; and simultaneously train the representation vector calculation model and the generation model according to each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample to obtain a trained representation vector calculation model and a trained generation model. Therefore, by simultaneously training the representation vector calculation model and the generation model, a multimodal learning advantage is established, a multi-angle understanding and application of address information is achieved, and the overall performance and application flexibility of the model are improved.
[0142] In order to clearly illustrate the above embodiment, an example is now given for illustration.
[0143] Figure 4 It is a schematic diagram of the principle of an address information processing method provided in an embodiment of the present application.
[0144] like Figure 4 As shown, the address information processing method may include the following modules:
[0145] 1. Approximate Error Retrieval Module
[0146] The purpose of this module is to retrieve and match address information close to the input address in the event of input errors. It contains the following submodules:
[0147] 1.1 Substring cutting module
[0148] This module is responsible for cutting any first address information in the address information database into all possible word segmentation substrings (first training substrings). For example, "China Post Savings Bank" can be cut into "China", "China Post", "Post", "Savings", "Postal Savings", "Savings Bank", "Postal Savings Bank" and other name segments that may exist independently in logic.
[0149] As a possible implementation method, the substring cutting module can use a string processing algorithm combined with the word segmentation method in NLP technology to ensure that the cutting results can cover all possible word segmentation substrings.
[0150] 1.2 Random Screening Module
[0151] This module is responsible for randomly selecting one candidate substring (candidate training substring) from all the segmentation substrings (first training substrings) obtained by cutting any first address information, and at the same time, constructing a batch (training batch) to ensure that all substrings in the batch (a training batch is generated based on the candidate training substring and the second training substring similar to the candidate training substring) are relatively similar, providing a basis for subsequent error generation and iterative training.
[0152] As a possible implementation method, the random screening module can use custom word segmentation to construct an inverted index for Chinese characters and pinyin characters contained in all substrings, and then use the TF-IDF score to retrieve substrings similar to the candidate substrings.
[0153] 2. Error generation module
[0154] This module is responsible for simulating possible errors in user input, such as typos and homophone errors, for the substrings in each batch.
[0155] As a possible implementation, the error generation module generates errors based on rule-based error simulation methods, such as replacing characters with their homophones, randomly inserting or deleting characters, etc. At the same time, some common error types in actual language use can also be considered.
[0156] 3. Deep learning training module
[0157] Retrieval training: Use the substrings and wrong versions of the substrings (adversarial samples) in each batch to train the representation vector calculation model. During the training process, the representation vector calculation model is optimized by calculating the cross entropy loss and cosine similarity so that the cosine similarity between the adversarial sample and the original correct sample in the feature space is the highest.
[0158] Generative training: Train the generative model to recover the correct address information from incorrect input.
[0159] Loss function design: Combine similarity-based loss and Rogue score-based generation loss to optimize model performance and ensure the accuracy and similarity of the output. That is, the loss function consists of two parts: one is similarity-based loss, which is used to let the model learn how to match incorrect input with the correct POI name; the other is Rogue score-based generation loss, which is used to optimize the performance of the generation module and ensure that the generated output is as close to the correct address information as possible.
[0160] 4. Prediction Module
[0161] 4.1. Use the approximate error retrieval module to retrieve multiple second substrings that are similar to the second address information to be queried input by the user from multiple first substrings obtained by cutting the second address information in the address information database.
[0162] 4.2. Use the trained representation vector calculation model to generate the representation vector of the second address information and the representation vector of each second substring, and find the third substring with the highest cosine similarity from the cosine similarity of the representation vector of the second address information and the representation vector of each second substring in the feature space.
[0163] 4.3. Determine whether the highest cosine similarity meets the preset threshold to determine the output result.
[0164] 4.4. If the highest cosine similarity does not reach the preset threshold, it indicates that the second address information queried by the user is not in the address information database, that is, the second address information entered by the user is incorrect. In this case, the trained generation model is used to correct the second address information and provide the user with possible correct address information (the content of the second address information after error correction);
[0165] 4.5 If the highest cosine similarity reaches a preset threshold, it indicates that the first address information corresponding to the third substring may be the address information that the user wants to query, and the first address information is provided to the user.
[0166] In summary, this application proposes a method based on TF-IDF rough recall, deep learning vector calculation, and cosine similarity retrieval, plus a method for generating a backup to handle speech input errors, which has the following advantages:
[0167] Reduce complexity and improve efficiency: By using the TF-IDF-based fast retrieval and recall module for initial screening, the amount of data that needs to be processed by the deep learning model is reduced, thereby reducing the computational complexity and increasing the processing speed, which can significantly improve the user experience.
[0168] Enhance model generalization ability: By generating adversarial samples and adding them to training, the model can learn more real-world changes and noise, enhancing the model's generalization and error correction capabilities for unseen error samples.
[0169] Improve error correction accuracy: Combining deep learning models to fine-tune the results of preliminary recall can further improve the accuracy of error correction. Deep learning models can capture more complex language features and contextual information, thereby more accurately identifying and correcting errors.
[0170] Strengthen the model's tolerance to voice input errors: Considering that voice input often has more uncertainty and errors (such as accent differences, unclear pronunciation, etc.), this application specifically optimizes the model's processing and correction of such input, significantly improving the accuracy of voice search and user satisfaction.
[0171] Dynamic adaptation and continuous optimization: The model can dynamically adapt to changes in language usage and emerging POI names by constantly learning new error patterns and correct correction methods, keeping the system continuously optimized and updated.
[0172] Advantages of multimodal learning: By performing retrieval training and generation training simultaneously, the model not only learns the ability to process text content, but also enhances the ability to generate from errors to correctness, realizes multi-angle understanding and application of text data, and improves the overall performance and application flexibility of the model.
[0173] In order to implement the above embodiments, the present application also proposes an address information processing device.
[0174] Figure 5 It is a structural diagram of an address information processing device provided in an embodiment of the present application.
[0175] like Figure 5 As shown, the address information processing device 500 includes: a reading module 501 , a first determining module 502 , a first generating module 503 and a second determining module 504 .
[0176] The reading module 501 is used to read the first address information in the address information database in response to the second address information to be queried input by the user, wherein the first address information is generated based on a preset standard address description;
[0177] A first determining module 502, configured to determine, from the plurality of first substrings obtained by cutting the first address information, a plurality of second substrings that are close to the second address information according to the degree of similarity between the plurality of first substrings obtained by cutting the first address information and the second address information;
[0178] A first generating module 503, configured to generate a representation vector of the second address information and a representation vector of each second substring by using a trained representation vector calculation model;
[0179] The second determination module 504 is used to determine whether to use the trained generation model to correct the second address information according to the cosine similarity between the characterization vector of the second address information and the characterization vectors of each second substring in the feature space.
[0180] In one embodiment of the present application, the first determining module 502 includes:
[0181] A construction unit is used to construct an inverted index for the first language characters included in the plurality of first substrings and the first symbol characters representing the pronunciation of the first language characters corresponding to each first language character, so as to obtain an inverted index corresponding to each first language character and an inverted index corresponding to each first symbol character, wherein the inverted index corresponding to each first language character records the substring identifiers of the fourth substrings in the first substrings that contain the first language characters and the occurrence frequency of the first language characters in the fourth substrings, and the inverted index corresponding to each first symbol character records the substring identifiers of the fifth substrings in the first substrings that contain the first symbol characters and the occurrence frequency of the first symbol characters in the fifth substrings;
[0182] a determining unit, configured to determine a term frequency-inverse document frequency TF-IDF score corresponding to at least one target substring including the second language character and / or the second symbol character among the plurality of first substrings based on an inverted index corresponding to each first language character, an inverted index corresponding to each first symbol character, the second language character included in the second address information, and a second symbol character representing a pronunciation of the second language character corresponding to each second language character;
[0183] The processing unit is used to sort the target substrings according to the TF-IDF scores corresponding to the target substrings, so as to determine the second substring according to the sorted target substrings.
[0184] In one embodiment of the present application, a construction unit is used to:
[0185] For any first substring among the plurality of first substrings, determining the occurrence frequency of a first language character included in the first substring in the first substring, and the occurrence frequency of a first symbol character corresponding to the first language character included in the first substring in the first substring;
[0186] Constructing an inverted index corresponding to each first language character according to the occurrence frequency of the first language character contained in each first substring in each first substring and the substring identifier of each first substring;
[0187] According to the occurrence frequency of the first symbol character corresponding to the first language character included in each first substring in each first substring and the substring identifier of each first substring, an inverted index corresponding to each first symbol character is constructed.
[0188] In one embodiment of the present application, the determining unit is configured to:
[0189] For any second language character included in the second address information, determine the appearance frequency of the second language character in the second address information, and the appearance frequency of a second symbol character corresponding to the second language character in the second address information;
[0190] In a case where any first language character is a second language character included in the second address information, determining at least one fourth substring recorded in the inverted index corresponding to the first language character as a target substring;
[0191] In the case where any first symbol character is a second symbol character corresponding to the second language character included in the second address information, determining at least one fifth substring recorded in the inverted index corresponding to the first symbol character as a target substring;
[0192] The TF-IDF score corresponding to each target substring is determined based on the occurrence frequency of the second language character in each target substring, the occurrence frequency of the second symbol character corresponding to the second language character in each target substring, the occurrence frequency of the second language character in the second address information, and the occurrence frequency of the second symbol character corresponding to the second language character in the second address information.
[0193] In one embodiment of the present application, the determining unit is configured to:
[0194] Initialize the TF-IDF score corresponding to each target substring to a preset value;
[0195] sequentially determining whether the frequency of occurrence of the second language character in each target substring is consistent with the frequency of occurrence of the second language character in the second address information, and if consistent, increasing the TF-IDF score corresponding to the target substring by a first set value;
[0196] It is determined in turn whether the frequency of occurrence of the second symbol character corresponding to the second language character in each target substring is consistent with the frequency of occurrence of the second symbol character corresponding to the second language character in the second address information, so as to increase the TF-IDF score corresponding to the target substring by a second set value if they are consistent.
[0197] In one embodiment of the present application, the processing unit is used to:
[0198] According to the TF-IDF scores corresponding to the target substrings, the target substrings are sorted in a set sorting order to obtain sorted target substrings;
[0199] When the sorting order is set to ascending, the target substring with the last set number of bits in each sorted target substring is determined as the second substring;
[0200] When the sorting order is set to descending, the first set number of target substrings in each sorted target substring is determined as the second substring.
[0201] In one embodiment of the present application, the device further includes a first processing module and a second processing module.
[0202] The first processing module is used to perform word segmentation processing on any first substring to obtain at least one first language character contained in the first substring;
[0203] The second processing module is used to perform conversion processing on each first language character to obtain a first symbol character corresponding to each first language character and representing the pronunciation of the first language character.
[0204] In one embodiment of the present application, the device further includes a cutting module, a third determining module, a second generating module and a training module.
[0205] The cutting module is used to cut the first address information for any first address information in the address information database to obtain a plurality of first training substrings corresponding to the first address information;
[0206] a third determination module, configured to determine a candidate training substring from the plurality of first training substrings, and determine a plurality of second training substrings similar to the candidate training substring from the target training substring according to the degree of similarity between the target training substring and the candidate training substring, wherein the target training substring is each first training substring in the plurality of first training substrings except the candidate training substring;
[0207] A second generating module, used for generating a training batch corresponding to the first address information according to the candidate training substrings and the plurality of second training substrings;
[0208] A training module is used to simultaneously train a representation vector calculation model and a generation model according to each training sample in at least one training batch corresponding to each first address information and an adversarial sample corresponding to each training sample, so as to obtain a trained representation vector calculation model and a trained generation model.
[0209] In one embodiment of the present application, the second generation module is used to:
[0210] For the candidate training substring and the plurality of second training substrings, a preset error generation rule is used to generate an adversarial sample corresponding to the candidate training substring and an adversarial sample corresponding to each second training substring;
[0211] The candidate training substring and the multiple second training substrings, as well as the adversarial sample corresponding to the candidate training substring and the adversarial sample corresponding to each second training substring, are determined as a training batch corresponding to the first address information.
[0212] In one embodiment of the present application, the training module is used to:
[0213] Input each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample into a representation vector calculation model, so that the cosine similarity in the feature space between the representation vector of any training sample generated by the trained representation vector calculation model and the representation vector of the adversarial sample corresponding to the training sample is higher than the cosine similarity in the feature space between the representation vector of the training sample and the representation vector of the adversarial sample corresponding to any training sample other than the training sample, wherein the representation vector calculation model uses a preset loss function to identify the correspondence between each training sample in any training batch and the adversarial sample corresponding to each training sample;
[0214] For any training batch corresponding to each first address information, the adversarial sample corresponding to each training sample in the training batch is used as the input of the generation model, and each training sample in the training batch is used as the output of the generation model. The generation model is trained so that the trained generation model can restore the adversarial sample corresponding to any training sample to the training sample.
[0215] In one embodiment of the present application, the third determination module is used to:
[0216] Constructing an inverted index for the third language characters included in the target training substring and the third symbol characters corresponding to each third language character and indicating the pronunciation of the third language character, so as to obtain an inverted index corresponding to each third language character and an inverted index corresponding to each third symbol character, wherein the inverted index corresponding to each third language character records a substring identifier of a first target training substring including the third language character in the target training substring and an occurrence frequency of the third language character in the first target training substring, and the inverted index corresponding to each third symbol character records a substring identifier of a second target training substring including the third symbol character in the target training substring and an occurrence frequency of the third symbol character in the second target training substring;
[0217] Determining a TF-IDF score corresponding to at least one third target training substring containing the fourth language character and / or the fourth symbol character in the target training substring according to an inverted index corresponding to each third language character, an inverted index corresponding to each third symbol character, a fourth language character contained in the candidate training substring, and a fourth symbol character representing a pronunciation of the fourth language character corresponding to each fourth language character;
[0218] The third target training substrings are sorted according to the TF-IDF scores corresponding to the third target training substrings, so as to determine the second training substrings according to the sorted third target training substrings.
[0219] In one embodiment of the present application, the second determining module is used to:
[0220] Determine, from each second substring, a third substring having the highest cosine similarity according to the cosine similarity between the second address information and each second substring;
[0221] When the cosine similarity between the second address information and the third substring does not reach a preset threshold, the second address information is corrected using the trained generation model;
[0222] When the cosine similarity between the second address information and the third substring reaches a preset threshold, the first address information corresponding to the third substring is provided to the user.
[0223] It should be noted that the above explanation of the embodiment of the address information processing method is also applicable to the address information processing device of this embodiment, and will not be repeated here.
[0224] The address information processing device provided by the present application reads the first address information in the address information database in response to the user inputting the second address information to be queried, wherein the first address information is generated based on a preset standard address description; according to the similarity between the multiple first substrings obtained by cutting the first address information and the second address information, multiple second substrings similar to the second address information are determined from the multiple first substrings; a representation vector of the second address information and the representation vectors of each second substring are generated by using a trained representation vector calculation model; according to the cosine similarity between the representation vector of the second address information and the representation vector of each second substring in the feature space, it is determined whether to use the trained generation model to correct the second address information. Thus, by preliminarily screening the multiple first substrings based on the similarity between the multiple first substrings obtained by cutting each first address information in the address information database and the second address information input by the user, the target range can be quickly narrowed down in large-scale data, the amount of data that needs to be processed by the deep learning model can be reduced, the computational complexity can be reduced, and the processing efficiency and user experience can be improved.
[0225] In an exemplary embodiment, an electronic device is also provided.
[0226] Among them, electronic equipment includes:
[0227] processor;
[0228] a memory for storing processor-executable instructions;
[0229] The processor is configured to execute instructions to implement the address information processing method proposed in any of the aforementioned embodiments.
[0230] As an example, Figure 6 is a schematic diagram of the structure of an electronic device 600 shown in an exemplary embodiment of the present application. Figure 6 As shown, the electronic device 600 may further include:
[0231] The memory 610 and the processor 620, a bus 630 connecting different components (including the memory 610 and the processor 620), the memory 610 stores a computer program, and when the processor 620 executes the program, the address information processing method described in the embodiment of the present application is implemented.
[0232] Bus 630 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.
[0233] The electronic device 600 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by the electronic device 600, including volatile and non-volatile media, removable and non-removable media.
[0234] The memory 610 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 640 and / or cache memory 650. The server 600 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 660 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, often referred to as a "hard drive packager"). Although Figure 6 Not shown, a disk driver package for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk driver package for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each driver package can be connected to bus 630 via one or more data medium interfaces. Memory 610 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of each embodiment of the present application.
[0235] A program / utility 680 having a set (at least one) of program modules 670 may be stored, for example, in the memory 610, such program modules 670 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 670 generally perform the functions and / or methods of the embodiments described herein.
[0236] The electronic device 600 may also communicate with one or more external devices 660 (e.g., keyboard, pointing device, display 691, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via an input / output (I / O) interface 692. In addition, the electronic device 600 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via a network adapter 693. As shown, the network adapter 693 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device driver package, redundant processing unit, external disk driver package array, RAID system, tape driver package, and data backup storage system, etc.
[0237] The processor 620 executes various functional applications and data processing by running the programs stored in the memory 610 .
[0238] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the address information processing method of the embodiment of the present application, and will not be repeated here.
[0239] In an exemplary embodiment, a computer-readable storage medium including instructions is also proposed, such as a memory including instructions, and the above instructions can be executed by a processor of an electronic device to complete the address information processing method proposed in any of the above embodiments. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0240] In an exemplary embodiment, a computer program product is also proposed, including a computer program / instruction, and when the computer program / instruction is executed by a processor, the address information processing method proposed in any of the above embodiments is implemented.
[0241] The collection, storage, access, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.
[0242] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and not shared or sold outside of these legitimate accesses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice before the user accesses the function, and signing the agreement / authorization including authorization of relevant user information. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.
[0243] This application is expected to provide an implementation scheme for users to selectively block access to or access to personal information data. That is, this application is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, risks can be minimized by limiting data collection and deleting data. In addition, when applicable, personal identifiers are removed from such personal information to protect the privacy of users.
[0244] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0245] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0246] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0247] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0248] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or accessed as an independent product, it may also be stored in a computer-readable storage medium.
[0249] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A method for processing address information, It is characterized in that include: In response to a user inputting second address information to be queried, reading first address information in an address information database, wherein the first address information is generated based on a preset standard address description; Determine, from the plurality of first substrings obtained by cutting the first address information, a plurality of second substrings that are close to the second address information according to the degree of similarity between the plurality of first substrings obtained by cutting the first address information and the second address information; Using the trained representation vector calculation model, generate a representation vector of the second address information and a representation vector of each of the second substrings; According to the cosine similarity between the characterization vector of the second address information and the characterization vectors of each second substring in the feature space, it is determined whether to use the trained generation model to perform correction processing on the second address information.
2. The method according to claim 1, It is characterized in that The similarity between the plurality of first substrings obtained by cutting the first address information and the second address information, and determining a plurality of second substrings close to the second address information from the plurality of first substrings, comprises: An inverted index is constructed for the first language characters included in the multiple first substrings and the first symbol characters representing the pronunciation of the first language characters corresponding to each of the first language characters, so as to obtain an inverted index corresponding to each of the first language characters and an inverted index corresponding to each of the first symbol characters, wherein the inverted index corresponding to each of the first language characters records a substring identifier of a fourth substring of the multiple first substrings including the first language character and an occurrence frequency of the first language character in the fourth substring, and the inverted index corresponding to each of the first symbol characters records a substring identifier of a fifth substring of the multiple first substrings including the first symbol character and an occurrence frequency of the first symbol character in the fifth substring; Determining a term frequency-inverse document frequency TF-IDF score corresponding to at least one target substring of the plurality of first substrings including the second language character and / or the second symbol character according to an inverted index corresponding to each of the first language characters, an inverted index corresponding to each of the first symbol characters, a second language character included in the second address information, and a second symbol character representing a pronunciation of the second language character corresponding to each of the second language characters; The target substrings are sorted according to the TF-IDF scores corresponding to the target substrings, so as to determine the second substring according to the sorted target substrings.
3. The method according to claim 2, It is characterized in that The step of constructing an inverted index for the first language characters included in the plurality of first substrings and the first symbol characters corresponding to each of the first language characters and representing the pronunciation of the first language characters, so as to obtain an inverted index corresponding to each of the first language characters and an inverted index corresponding to each of the first symbol characters, comprises: For any first substring among the plurality of first substrings, determining an occurrence frequency of a first language character included in the first substring in the first substring, and an occurrence frequency of a first symbol character corresponding to the first language character included in the first substring in the first substring; constructing an inverted index corresponding to each first language character according to the occurrence frequency of the first language character contained in each first substring in each first substring and the substring identifier of each first substring; An inverted index corresponding to each first symbol character is constructed according to the occurrence frequency of the first symbol character corresponding to the first language character included in each first substring in each first substring and the substring identifier of each first substring.
4. The method according to claim 2, It is characterized in that The determining, according to the inverted index corresponding to each of the first language characters, the inverted index corresponding to each of the first symbol characters, the second language characters included in the second address information, and the second symbol characters representing the pronunciation of the second language characters corresponding to each of the second language characters, a term frequency-inverse document frequency TF-IDF score corresponding to at least one of the plurality of first substrings including the second language characters and / or the second symbol characters, comprises: For any second language character included in the second address information, determine the occurrence frequency of the second language character in the second address information, and the occurrence frequency of a second symbol character corresponding to the second language character in the second address information; In the case where any of the first language characters is a second language character included in the second address information, determining at least one of the fourth substrings recorded in the inverted index corresponding to the first language character as the target substring; In the case where any of the first symbol characters is a second symbol character corresponding to a second language character included in the second address information, determining at least one of the fifth substrings recorded in the inverted index corresponding to the first symbol character as the target substring; The TF-IDF score corresponding to each target substring is determined based on the occurrence frequency of the second language character in each target substring, the occurrence frequency of the second symbol character corresponding to the second language character in each target substring, the occurrence frequency of the second language character in the second address information, and the occurrence frequency of the second symbol character corresponding to the second language character in the second address information.
5. The method according to claim 4, It is characterized in that Determining the TF-IDF score corresponding to each of the target substrings according to the occurrence frequency of the second language character in each of the target substrings, the occurrence frequency of the second symbol character corresponding to the second language character in each of the target substrings, the occurrence frequency of the second language character in the second address information, and the occurrence frequency of the second symbol character corresponding to the second language character in the second address information includes: Initialize the TF-IDF score corresponding to each target substring to a preset value; sequentially determining whether the occurrence frequency of the second language character in each of the target substrings is consistent with the occurrence frequency of the second language character in the second address information, and if consistent, increasing the TF-IDF score corresponding to the target substring by a first set value; Determine in turn whether the frequency of occurrence of the second symbol character corresponding to the second language character in each of the target substrings is consistent with the frequency of occurrence of the second symbol character corresponding to the second language character in the second address information, and if they are consistent, increase the TF-IDF score corresponding to the target substring by a second set value.
6. The method according to claim 2, It is characterized in that The step of sorting the target substrings according to the TF-IDF scores corresponding to the target substrings to determine the second substring according to the sorted target substrings includes: According to the TF-IDF scores corresponding to the target substrings, the target substrings are sorted in a set sorting order to obtain sorted target substrings; In the case where the set sorting order is ascending sorting, the last set number of bits in each of the sorted target substrings is determined as the second substring; In the case where the set sorting order is descending sorting, the first set number of digits of each of the sorted target substrings is determined as the second substring.
7. The method according to claim 2, It is characterized in that Before constructing an inverted index for the first language characters included in the plurality of first substrings and the first symbol characters representing the pronunciation of the first language characters corresponding to each of the first language characters to obtain an inverted index corresponding to each of the first language characters and an inverted index corresponding to each of the first symbol characters, the method further includes: Performing word segmentation processing on any of the first substrings to obtain at least one first language character contained in the first substring; The first language characters are converted to obtain first symbol characters corresponding to each first language character and representing the pronunciation of the first language character.
8. The method according to claim 1, It is characterized in that The training process of the representation vector calculation model and the generation model includes: For any first address information in the address information database, segment the first address information to obtain a plurality of first training substrings corresponding to the first address information; Determine a candidate training substring from the multiple first training substrings, and determine a plurality of second training substrings similar to the candidate training substring from the target training substring according to the similarity between the target training substring and the candidate training substring, wherein the target training substring is each of the first training substrings in the multiple first training substrings except the candidate training substring; Generating a training batch corresponding to the first address information according to the candidate training substring and the plurality of second training substrings; According to each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample, the representation vector calculation model and the generation model are trained simultaneously to obtain the trained representation vector calculation model and the trained generation model.
9. The method according to claim 8, It is characterized in that The step of generating a training batch corresponding to the first address information according to the candidate training substring and the plurality of second training substrings includes: For the candidate training substring and the multiple second training substrings, using a preset error generation rule, generate an adversarial sample corresponding to the candidate training substring and an adversarial sample corresponding to each of the second training substrings; The candidate training substring and the multiple second training substrings, as well as the adversarial sample corresponding to the candidate training substring and the adversarial sample corresponding to each of the second training substrings, are determined as a training batch corresponding to the first address information.
10. The method according to claim 8, It is characterized in that The method of simultaneously training the representation vector calculation model and the generation model according to each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample to obtain the trained representation vector calculation model and the trained generation model includes: Input each training sample in at least one training batch corresponding to each first address information and the adversarial sample corresponding to each training sample into the representation vector calculation model, so that the cosine similarity in the feature space between the representation vector of any training sample generated by the trained representation vector calculation model and the representation vector of the adversarial sample corresponding to the training sample is higher than the cosine similarity in the feature space between the representation vector of the training sample and the representation vector of the adversarial sample corresponding to any training sample other than the training sample, wherein the representation vector calculation model uses a preset loss function to identify the correspondence between each training sample in any training batch and the adversarial sample corresponding to each training sample; For any training batch corresponding to each of the first address information, the adversarial sample corresponding to each training sample in the training batch is used as the input of the generation model, and each training sample in the training batch is used as the output of the generation model, and the generation model is trained so that the trained generation model can restore the adversarial sample corresponding to any of the training samples to the training sample.
11. The method according to claim 8, It is characterized in that The step of determining a plurality of second training substrings that are similar to the candidate training substring from the target training substring according to the similarity between the target training substring and the candidate training substring comprises: Constructing an inverted index for the third language characters contained in the target training substring and the third symbol characters corresponding to each of the third language characters and indicating the pronunciation of the third language characters, so as to obtain an inverted index corresponding to each of the third language characters and an inverted index corresponding to each of the third symbol characters, wherein the inverted index corresponding to each of the third language characters records a substring identifier of a first target training substring containing the third language character in the target training substring and an occurrence frequency of the third language character in the first target training substring, and the inverted index corresponding to each of the third symbol characters records a substring identifier of a second target training substring containing the third symbol character in the target training substring and an occurrence frequency of the third symbol character in the second target training substring; Determining a TF-IDF score corresponding to at least one third target training substring containing the fourth language character and / or the fourth symbol character in the target training substring according to an inverted index corresponding to each of the third language characters, an inverted index corresponding to each of the third symbol characters, a fourth language character contained in the candidate training substring, and a fourth symbol character representing a pronunciation of the fourth language character corresponding to each of the fourth language characters; The third target training substrings are sorted according to the TF-IDF scores corresponding to the third target training substrings, so as to determine the second training substrings according to the sorted third target training substrings.
12. The method according to any one of claims 1 to 11, It is characterized in that The determining, according to the cosine similarity between the second address information and each of the second substrings, whether to use the trained generation model to perform correction processing on the second address information includes: Determine, from each of the second substrings, a third substring having the highest cosine similarity according to the cosine similarity between the second address information and each of the second substrings; When the cosine similarity between the second address information and the third substring does not reach a preset threshold, modifying the second address information using a trained generation model; When the cosine similarity between the second address information and the third substring reaches a preset threshold, the first address information corresponding to the third substring is provided to the user.
13. An address information processing device, It is characterized in that include: A reading module, configured to read first address information in an address information database in response to second address information to be queried input by a user, wherein the first address information is generated based on a preset standard address description; A first determining module, configured to determine, from the plurality of first substrings obtained by cutting the first address information, a plurality of second substrings that are close to the second address information according to the degree of similarity between the plurality of first substrings obtained by cutting the first address information and the second address information; A first generating module, configured to generate a representation vector of the second address information and a representation vector of each second substring by using a trained representation vector calculation model; The second determination module is used to determine whether to use the trained generation model to correct the second address information according to the cosine similarity between the representation vector of the second address information and the representation vector of each second substring in the feature space.
14. An electronic device, It is characterized in that include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 12.
15. A computer-readable storage medium, It is characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 12.
16. A computer program product, It is characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 12 when being executed by a processor.
Citation Information
Patent Citations
Address text recognition method and device, medium and electronic equipment
CN111695355A
Address standardization method and system and electronic equipment
CN112417812A
Address matching method and device, electronic equipment and storage medium
CN112818685A
Address entity error correction method and device and computer readable storage medium
CN112966495A
Address information correction method and device, computer equipment and storage medium
CN113505190A
