Method, electronic device and medium for automatically extracting travel diary data

Abstract

The present invention discloses a method, electronic device and medium for automatically extracting travel notes data, wherein the method for automatically extracting travel notes data comprises S100, performing word segmentation and naming body recognition on the text to be processed to obtain multiple word segments and multiple naming bodies; S200, screening candidate destinations from multiple word segments based on a destination dictionary table, and screening candidate destinations from multiple naming bodies based on a corpus; S300, if the number of the candidate destinations is greater than a preset value, clustering all candidate destinations to obtain multiple cluster clusters; S400, taking the candidate destination corresponding to the cluster with the most cluster elements as the final destination. The present invention realizes accurate recognition of the destinations and POIs mentioned in the user's travel notes through the method for automatically extracting travel notes data, and improves the recognition accuracy of travel notes keywords.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a method, electronic device and medium for automatically extracting travel diary data. Background Art

[0002] In recent years, during the development of the industry, a batch of excellent UGC (User-generated Content) travel notes, travel photos, reviews and other text data output by users have been accumulated. Most of these data are unstructured data. If they are not structured to extract key information points, it will be difficult to apply them to subsequent scenarios such as multiple complex text extraction POI (Point of Interest), such as travel notes details page mining and displaying keywords, automatic travel notes listing, POI information link display, enhanced SEO (Search Engine Optimization) assistance, assisted manual review, automatic itinerary generation, etc.

[0003] It is necessary to automatically extract the main destinations and POIs involved in UGC data, automatically correct fuzzy matching identification, and automatically link to the knowledge base. First of all, if you do not have rich industry knowledge and know the names of destinations and POIs well, it is difficult to distinguish the destination and POI information mentioned in the travel notes. Secondly, the destinations and POIs mentioned in travel notes are rich and diverse, and there are many of them. If the destinations and POIs are extracted manually, it will take a huge amount of manpower, material resources, time and financial resources. It is difficult to quickly and accurately distinguish the relationship between the large number of POIs in the database and the extracted destinations and POIs. Moreover, due to the inherent characteristics of the manual review method, the information that needs to be extracted may be missed or errors may be made due to fatigue or carelessness. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defect in the prior art that the accuracy of manually extracting destinations and POIs needs to be improved, and to provide a method, electronic device and medium for automatically extracting travel diary data.

[0005] The present invention solves the above technical problems through the following technical solutions:

[0006] A method for automatically extracting travel diary data, the method comprising the following steps:

[0007] S100, performing word segmentation and named body recognition on the text to be processed to obtain multiple word segments and multiple named bodies;

[0008] S200, filtering out candidate destinations from the multiple word segments based on a destination dictionary table, and filtering out candidate destinations from the multiple named entities based on a corpus;

[0009] S300, if the number of the candidate destinations is greater than a preset value, clustering all the candidate destinations to obtain a plurality of clusters;

[0010] S400: The candidate destination corresponding to the cluster with the most cluster elements is taken as the final destination.

[0011] Preferably, in step S300, clustering is performed based on a K-means clustering algorithm.

[0012] Preferably, before step S200, the following steps are further included:

[0013] S101 . Generate a Chinese stop word dictionary according to commonly used Chinese stop words, and filter the plurality of segmented words according to the Chinese stop word dictionary.

[0014] Preferably, step S100 specifically includes:

[0015] The Jieba toolkit based on Markov chain performs word segmentation and named body recognition on the text to be processed to obtain multiple word segments and multiple named bodies.

[0016] Preferably, step S100 further includes:

[0017] Performing part-of-speech tagging on the multiple participles;

[0018] Step S200 specifically includes:

[0019] S201: Filter out the candidate destinations from the participles marked as nouns based on the destination dictionary table.

[0020] Preferably, the method for automatically extracting travel diary data further includes:

[0021] S500, constructing a POI dictionary table, wherein the POI dictionary table includes all POIs under the destination directory in the destination dictionary table;

[0022] S600: Match candidate POIs from the multiple word segments and the multiple named entities based on the POI dictionary table.

[0023] Preferably, the POI includes a plurality of POI entities, and the POI entity includes a name, an alias, an English name, coordinates, a type, a destination, a destination path, and a quality score;

[0024] Step S600 specifically includes:

[0025] If a plurality of candidate POIs correspond to the same POI entity, the POI entity with the longest name is retained.

[0026] Preferably, step S600 specifically includes:

[0027] Based on the POI dictionary table, the segmented words and the named entities containing the POI are matched from the multiple segmented words and the multiple named entities, and the candidate POI is matched according to the coordinates of the POI and the coordinates of the destination.

[0028] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for automatically extracting travel data of the present invention when executing the computer program.

[0029] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for automatically extracting travel data of the present invention are implemented.

[0030] The positive and progressive effects of the present invention are: by automatically extracting multiple destinations and POIs involved in UGC travel text data, automatically correcting fuzzy matching recognition, filtering out interference information, and improving the recognition accuracy of travel text keywords; the extracted data can be subsequently applied to the details page of travel notes to mine and display keywords, automatically hang goods in travel notes, display POI information links, enhance SEO assistance, assist manual review, and automatically generate itineraries. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of a method for automatically extracting travel diary data according to Embodiment 1 of the present invention.

[0032] Figure 2 This is a flow chart of a preferred solution of the method for automatically extracting travel diary data according to Example 1 of the present invention.

[0033] Figure 3 This is a schematic diagram of a module of an electronic device according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0035] Example 1

[0036] This embodiment provides a method for automatically extracting travel diary data. Figure 1 As shown, the automatic data extraction method includes:

[0037] S100, performing word segmentation and named body recognition on the text to be processed to obtain multiple word segments and multiple named bodies;

[0038] Specifically, the Jieba toolkit based on Markov chain is used here to perform word segmentation and named entity recognition on travel notes data. The Jieba toolkit can realize efficient word graph scanning through prefix dictionary, and generate a directed acyclic graph consisting of all possible word formation situations of Chinese characters in the input text. The dynamic programming algorithm is used to search for the maximum probability word path and find the maximum segmentation combination based on word frequency. Before word segmentation, the travel notes text can also be preprocessed and data cleaned. For example, regular expressions can be used to match non-Chinese, uppercase and lowercase, and other characters other than numbers, and replace them with spaces; regular expressions can be used to match various brackets and delete them; special character dictionaries and punctuation dictionary can be matched to delete them; full-width characters can be replaced with half-width characters through full-width and half-width comparison dictionaries; English letter dictionaries can be matched to mark characters; and digital dictionaries can be matched to replace them with Chinese characters.

[0039] S200, filtering out candidate destinations from the multiple word segments based on a destination dictionary table; the destination dictionary table includes a destination ID, a destination name, a destination type, a destination coordinate, and a city level corresponding to the destination;

[0040] S300, if the number of the candidate destinations is greater than a preset value, clustering all the candidate destinations to obtain a plurality of clusters;

[0041] S400, taking the candidate destination corresponding to the cluster with the most clustering elements as the final destination; filtering out the destination corresponding to the cluster with fewer clustering elements. The final destination can also be determined based on whether the distance between the candidate destination and the user's destination is greater than a preset distance. The final destination can be used for keyword mining and display on the details page of travel notes, automatic cargo hanging in travel notes, POI information link display, enhanced SEO assistance, auxiliary manual review, automatic itinerary generation, etc.

[0042] S500, constructing a POI dictionary table, the POI dictionary table including all POIs under the destination directory in the destination dictionary table;

[0043] POI includes multiple POI entities, including name, alias, English name, coordinates, type, destination, destination path, and quality score; according to the POI entity, the POI name is the key and the related information is the value.

[0044] S600, matching candidate POIs from multiple word segments and multiple named entities of the travel text based on the POI dictionary table;

[0045] Step S600 specifically includes: matching the segmentation and naming bodies containing POI from multiple segmentation and naming bodies based on the POI dictionary table, matching the candidate POI according to the coordinates of the POI and the coordinates of the destination; specifically, calculating the edit distance, Jaccard distance, positive ratio of the same number of words, negative ratio of the same number of words, and the longest common substring between each segmentation result and the Key in the POI dictionary; setting a threshold for filtering according to the non-independent characteristics of the content to obtain a candidate entity set corresponding to each segmentation result; rejecting all unknown entities that cannot enter the threshold range; determining a candidate set, recalling a batch of segmentation results according to the value of the longest common substring, and discarding the others; using the content-independent features POI quality score and POI type to sort all candidate results, and selecting the POI entity corresponding to the segmentation result with the highest POI quality score; matching the candidate POI according to the coordinates of the POI and the coordinates of the destination. If multiple candidate POIs correspond to the same POI entity, the POI entity with the longest name is retained. In addition, the positions of brackets in the text are calculated, and the length of the text in the brackets is less than the threshold. If the matched POI is covered by these bracket positions, the POI is discarded.

[0046] In addition, this embodiment also provides a preferred solution, such as Figure 2 As shown, before step S200, the following steps are also included:

[0047] S101. Generate a Chinese stop word dictionary based on commonly used Chinese stop words, and filter the segmented words based on the Chinese stop word dictionary, common word types, and common scene region dictionaries.

[0048] like Figure 2 As shown, the preferred solution provided by this embodiment is based on the method for automatically extracting travel notes data in this embodiment, and step S100 also includes:

[0049] Perform part-of-speech tagging on multiple words;

[0050] Step S200 specifically includes:

[0051] S201 : Filter out candidate destinations from the words marked as nouns based on the destination dictionary table.

[0052] Through the method for automatic extraction of travel note data provided in this embodiment, it is possible to accurately identify and screen the destinations and POIs mentioned in the travel note text, retain the main destinations and POIs in the travel note, and filter out interference information. The extracted destination and POI information can be used in auxiliary manual review and front-end display of travel photography, and large-scale keyword search and drainage scenarios, thereby improving the overall accuracy and coverage of keywords, facilitating the distribution of UGC content, and being able to greatly reduce the difficulty and time consumption of reviewers' work, and further saving the operating costs of the operating entity.

[0053] Example 2

[0054] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for automatically extracting travel data described in Embodiment 1 is implemented.

[0055] Figure 3 A schematic diagram of the structure of an electronic device provided in this embodiment. Figure 3 A block diagram of an exemplary electronic device 90 suitable for use in implementing embodiments of the present invention is shown. Figure 3 The electronic device 90 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0056] like Figure 3 As shown, the electronic device 90 may be in the form of a general-purpose computing device, for example, it may be a server device. The components of the electronic device 90 may include, but are not limited to: at least one processor 91, at least one memory 92, and a bus 93 connecting different system components (including the memory 92 and the processor 91).

[0057] The bus 93 includes a data bus, an address bus, and a control bus.

[0058] The memory 92 may include a volatile memory, such as a random access memory (RAM) 921 and / or a cache memory 922 , and may further include a read-only memory (ROM) 923 .

[0059] The memory 92 may also include a program tool 925 having a set (at least one) of program modules 924, such program modules 924 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0060] The processor 91 executes various functional applications and data processing by running computer programs stored in the memory 92 .

[0061] The electronic device 90 may also communicate with one or more external devices 94 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 95. Furthermore, the electronic device 90 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 96. The network adapter 96 communicates with other modules of the electronic device 90 via a bus 93. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 90, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.

[0062] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0063] Example 3

[0064] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for automatically extracting travel data described in Example 1.

[0065] The readable storage medium may include but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device or any suitable combination of the above.

[0066] In a possible implementation, the present invention can also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of the method for automatically extracting travel data described in Example 1.

[0067] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on a user device, partially on a user device, as an independent software package, partially on a user device and partially on a remote device, or entirely on a remote device.

Claims

1. A method for automatically extracting travel diary data, characterized in that: The method comprises the following steps: S100, performing word segmentation and named body recognition on the text to be processed to obtain multiple word segments and multiple named bodies; S200, filtering out candidate destinations from the multiple word segments based on a destination dictionary table, and filtering out candidate destinations from the multiple named entities based on a corpus; S300, if the number of the candidate destinations is greater than a preset value, clustering all the candidate destinations to obtain a plurality of clusters; S400, taking the candidate destination corresponding to the cluster with the most cluster elements as the final destination; The method for automatically extracting travel diary data also includes: S500, constructing a POI dictionary table, wherein the POI dictionary table includes all POIs under the destination directory in the destination dictionary table; S600: Match candidate POIs from the multiple word segments and the multiple named entities based on the POI dictionary table.

2. The method for automatically extracting travel notes data according to claim 1, characterized in that: In step S300, clustering is performed based on the K-means clustering algorithm.

3. The method for automatically extracting travel notes data according to claim 1, characterized in that: Before step S200, the following steps are also included: S101 . Generate a Chinese stop word dictionary according to commonly used Chinese stop words, and filter the multiple word segments according to the Chinese stop word dictionary.

4. The method for automatically extracting travel notes data according to claim 1, characterized in that: Step S100 specifically includes: The Jieba toolkit based on Markov chain performs word segmentation and named entity recognition on the text to be processed to obtain multiple word segments and multiple named entities.

5. The method for automatically extracting travel notes data according to claim 1, characterized in that: Step S100 also includes: Performing part-of-speech tagging on the multiple participles; Step S200 specifically includes: S201: Filter out the candidate destinations from the participles marked as nouns based on the destination dictionary table.

6. The method for automatically extracting travel notes data according to claim 1, characterized in that: The POI includes multiple POI entities, and the POI entity includes a name, an alias, an English name, coordinates, a type, a destination, a destination path, and a quality score; Step S600 specifically includes: If a plurality of candidate POIs correspond to the same POI entity, the POI entity with the longest name is retained.

7. The method for automatically extracting travel notes data according to claim 1, characterized in that: Step S600 specifically includes: Based on the POI dictionary table, the segmented words and the named entities containing the POI are matched from the multiple segmented words and the multiple named entities, and the candidate POI is matched according to the coordinates of the POI and the coordinates of the destination.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for automatically extracting travel data according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for automatically extracting travel data according to any one of claims 1 to 7 is implemented.

Images