User positioning-oriented geographic word vector construction method

Abstract

The invention provides a user positioning-oriented geographic word vector construction method, which belongs to the technical field of natural language processing, and specifically comprises the following steps: constructing a geographic corpus by taking a geographic entity and a text corpus associated with the geographic entity as data sources; the geographic corpus is input into the BERT model for training, a geographic word vector of each geographic word in the geographic corpus is obtained, and the geographic word vectors comprise context information of the geographic words in sentences; and training the Bi LSTM-CRF model based on the geographic word vector, and carrying out place name identification and extraction by using the trained Bi LSTM-CRF model. Through the processing scheme provided by the invention, the accuracy of geographic information extraction is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method for constructing geographic word vectors for user positioning. Background Art

[0002] The place name recognition method based on deep learning has attracted much attention. After training geographic word vectors on a large amount of corpus on a pre-trained model, it is input into a neural network model and learned to complete the task of geographic entity recognition. At present, the research on geographic word vector models mainly includes one-hot, Word2Vec and other models, but they all have the following problems: 1) they cannot connect the context information; 2) they cannot dynamically express the polysemy of words. Especially in the field of geography, there are some abbreviated place names or complex place names, which leads to the weak expression ability of place name vector features.

[0003] Deep learning technology based on neural networks introduces word vectors to represent words in text geographic information recognition. Word vectors are a method of learning and extracting word feature information from large-scale text data, which converts various contextual relationships of text into the form of mathematical vectors. Deep learning uses neural networks to learn features of geographic word vectors, which not only avoids complex manual extraction, but also has good generalization capabilities. In the early days, Hammerton et al. used neural networks to study NER technology and adopted a unidirectional long short-term memory network (LSTM) with excellent sequence modeling capabilities, so the LSTM-CRF model became the mainstream technology of NER. Guillaume Lample et al. borrowed their ideas and proposed a neural network model that combines a bidirectional long short-term memory network model (BiLSTM) and a conditional random field model (CRF), which can obtain bidirectional contextual relationships and has been widely used in subsequent geographic information recognition related tasks.

[0004] However, the above methods all have the following two problems: First, as the complexity of the task increases, the amount of data required to train the deep learning model also increases. Since the training data usually needs to be manually labeled, the cost of obtaining massive training data is very high, which makes training or improving the model a time-consuming and labor-intensive project, which is very unfavorable for the promotion and application of deep models. Secondly, the geographic word vector training model built in the current named entity recognition task in the geographical field cannot represent the phenomenon of polysemy, because these models only extract word features and do not consider the association between the context, resulting in the final training of static geographic word vectors, which reduces the ability to recognize geographic named entities.

[0005] To solve the first problem, people try to extract the common knowledge generated in the deep learning process and use it in related machine learning tasks. Based on this reuse idea, transfer learning divides a complete training task into two steps: pre-training and fine-tuning.

[0006] Pre-training stage: The training goal of this stage is to generate a model that contains reusable knowledge - a pre-training model. In order to benefit more different tasks, the pre-training model must contain as much reusable knowledge as possible, so the pre-training models are very complex. Such complex models can only be trained with massive amounts of data, and this stage consumes a lot of computing resources. Although it is very expensive, people will continue to invest a lot of resources in pre-training because it can be invested once and reused.

[0007] Fine-tuning stage: Based on the pre-trained model, design and add the corresponding fine-tuning task layer according to the needs of the task. The main purpose of this stage is to train the parameters of the fine-tuning task layer. Since the fine-tuning task layer is usually not very complex and has a relatively small number of parameters, less training data can be used to complete the training work.

[0008] To solve the second problem, Jacob Devlin and others from the Google team invented the BERT language preprocessing model based on transfer learning to characterize a single word vector. The BERT model is a language model trained on a text corpus of more than 3.3 billion words, and then fine-tuned according to different specified tasks. The model has achieved the best results so far in 11 different NLP tasks. Huang Zhiheng and others have experimentally proved that the BiLSTM-CRF model does have a good performance in sequence labeling tasks. Jason P.CChiu, Eric Nichols and others combined the BiLSTM model with the CRF network. The F1 value of this deep learning method is 2.3%, which is higher than the best result on the On-toNotes 5.0 dataset. Based on this model, it has now become an advanced model for tasks such as NER.

[0009] In summary, in the early days, text geographic information recognition mainly used rule matching and statistical methods, and in recent years, it mainly used transfer learning and deep learning methods. However, there are still problems such as single geographic word vector features and inability to accurately extract special place names. Currently, there are few fusion BERT models and place name recognition networks (BiLSTM-CRF models), especially in image geographic positioning. Summary of the invention

[0010] In view of this, an embodiment of the present application provides a method for constructing a geographic word vector for user positioning, which at least partially solves the problem in the prior art that the geographic word vector features are single and special place names cannot be accurately extracted.

[0011] The present application embodiment provides a method for constructing a geographic word vector for user positioning, including:

[0012] Use geographic entities and text corpora associated with them as data sources to build a geographic corpus;

[0013] Input the geographic corpus into the BERT model for training to obtain a geographic word vector for each geographic word in the geographic corpus, wherein the geographic word vector includes context information of the geographic word in the sentence;

[0014] The BiLSTM-CRF model is trained based on the geographic word vector, and the trained BiLSTM-CRF model is used to recognize and extract place names.

[0015] According to a specific implementation of the embodiment of the present application, the construction of a geographic corpus using geographic entities and text corpora associated with the geographic entities as data sources includes:

[0016] Use web crawler technology to obtain texts and news transcripts on tourism and food topics from websites, and then organize them to obtain raw text;

[0017] Perform data preprocessing on raw corpus text;

[0018] The preprocessed raw corpus text is integrated with the People's Daily annotated corpus to obtain the geographic corpus.

[0019] According to a specific implementation of the embodiment of the present application, the method further includes:

[0020] The geographic corpus is subjected to corpus preprocessing, wherein the corpus preprocessing includes cleaning, word segmentation and part-of-speech tagging.

[0021] According to a specific implementation of an embodiment of the present application, the cleaning method includes an arrangement method, a manual deduplication method, a marking method and a deletion method.

[0022] According to a specific implementation of the embodiment of the present application, the part-of-speech tagging adopts Jieba tool.

[0023] According to a specific implementation method of an embodiment of the present application, the training of the BERT model includes a pre-training stage and a fine-tuning stage. The pre-training stage is used to train unlabeled raw corpus, and the fine-tuning stage is used to adjust model parameters based on preset applications to train labeled corpus.

[0024] According to a specific implementation method of an embodiment of the present application, the pre-training stage includes masked language model training and next sentence prediction training. Through the masked language model training, the BERT model learns the representation of each geographic word in different contexts, and through the next sentence prediction training, the BERT model learns the relationship between sentences.

[0025] According to a specific implementation of an embodiment of the present application, the fine-tuning stage includes sentence pair classification training, single sentence classification training, question-answering training and sequence labeling training.

[0026] According to a specific implementation of the embodiment of the present application, the method further includes:

[0027] The evaluate function of the Kashgari library file is used to evaluate the model quality of the trained BiLSTM-CRF model.

[0028] According to a specific implementation of the embodiment of the present application, the evaluation indicators of the model quality include precision, recovery rate and F1 value, and the F1 value is a comprehensive value of precision and recovery rate. The calculation formula of each evaluation indicator is:

[0029]

[0030]

[0031]

[0032] Among them, P is the precision rate, R is the recovery rate, a is the number of place names correctly recognized by the BiLSTM-CRF model, A is the total number of place names, and B is the number of place names that have been recognized.

[0033] Beneficial effects:

[0034] The method for constructing geographic word vectors for user positioning in the embodiment of the present application improves the overall performance of geographic information recognition and positioning by constructing word vectors with strong geographic semantics, thereby solving the shortcomings of the prior art in terms of inaccurate semantic understanding in geographic information processing. Experiments have shown that the geographic corpus obtained by collating the general geographic corpus with the self-made corpus of this application has achieved the best results in terms of accuracy, recovery rate and F1 value. The accuracy of this application is 8 percentage points higher than that of the general geographic corpus and 4% higher than that of a separate self-made corpus. Therefore, this application can extract all relevant geographic names more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 is a schematic diagram of recognition results based on different corpora according to an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of an official website page of a news program according to an embodiment of the present invention;

[0038] Figure 3 is a schematic diagram of a date character string in a link address according to an embodiment of the present invention;

[0039] Figure 4 A schematic diagram of a crawling result of a news program web page according to an embodiment of the present invention;

[0040] Figure 5 A schematic diagram of crawling results of the "Yigecun" website according to an embodiment of the present invention;

[0041] Figure 6 A schematic diagram of a place name dictionary format according to an embodiment of the present invention;

[0042] Figure 7 Schematic diagram of sample data after sequence annotation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0044] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0045] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0046] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0047] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0048] The present application embodiment provides a method for constructing a geographic word vector for user positioning, which mainly includes the following steps:

[0049] Step 1: Use geographic entities and text corpora associated with geographic entities as data sources to build a geographic corpus;

[0050] Step 2: Input the geographic corpus into the BERT model for training to obtain a geographic word vector for each geographic word in the geographic corpus, wherein the geographic word vector includes context information of the geographic word in the sentence;

[0051] Step 3: Train the BiLSTM-CRF model based on the geographic word vector, and use the trained BiLSTM-CRF model to recognize and extract place names.

[0052] In one embodiment, the step of using geographic entities and text corpora associated with the geographic entities as data sources to construct a geographic corpus includes:

[0053] Use web crawler technology to obtain texts and news transcripts on tourism and food topics from websites, and then organize them to obtain raw text;

[0054] Perform data preprocessing on raw corpus text;

[0055] The preprocessed raw corpus text is integrated with the People's Daily annotated corpus to obtain the geographic corpus.

[0056] In one embodiment, the method further comprises:

[0057] The geographic corpus is subjected to corpus preprocessing, wherein the corpus preprocessing includes cleaning, word segmentation and part-of-speech tagging.

[0058] In the specific implementation, for step 1, the following steps are specifically included:

[0059] Step 1.1, corpus selection.

[0060] The text-based construction and design of general corpora (including the People's Daily Chinese Corpus, MSRA Chinese Corpus, etc.) have mature working methods. The corpus mainly studied in this application is a geographic corpus dedicated to the field of geography. The geographic corpus uses geographic entities or text corpora with a relatively large correlation with geographic entities as data sources. Under a unified geographic entity labeling system, geographic information labels are set for the data set through natural language processing tools to form a geographic corpus.

[0061] Step 1.2, build a geographic corpus.

[0062] The geographic corpus constructed in this application is a combination of the 1998 edition of the People's Daily annotated corpus and the self-made geographic text corpus. The 1998 edition of the People's Daily dataset is a corpus jointly developed and created by China Fujitsu Co., Ltd. and the Peking University Computer Research Center. Although the version of the corpus is relatively old, it can still be seen in various competitions and practices. It has always been used as a standard corpus for reference by professionals. There are more than 1.75 million words in the corpus, involving about 28,000 geographical nouns. In this application, the self-made geographic text corpus is obtained by web crawler technology to obtain the "travel" and "food" topic texts on the Sina Weibo website and the 2020 news scripts of the news program. After sorting out the crawling results, about 3.1 million words are obtained to form a raw corpus text, and it is integrated with the annotated corpus of the People's Daily after data preprocessing such as word segmentation and part-of-speech tagging, together as the data set of the applied geographic corpus.

[0063] Step 1.3, corpus preprocessing.

[0064] The steps of corpus preprocessing mainly include cleaning, word segmentation, part-of-speech tagging, etc.

[0065] Cleaning: Corpus cleaning refers to preserving relevant information and data in a corpus and discarding noise data, such as punctuation marks and other unnecessary characters. Common cleaning methods include permutation, manual deduplication, tagging, deletion, etc.

[0066] Word segmentation: Text segmentation is the next step of data cleaning, which separates the text into words. Word segmentation is the basis for implementing natural language processing (NLP) tasks. The quality of word segmentation can even directly determine the accuracy of the subsequent semantic analysis results. Commonly used word segmentation techniques mainly include statistical-based word segmentation methods and rule-based word segmentation methods. Among them, the statistical-based word segmentation method calculates the joint distribution probability corresponding to each word segmentation method, and then selects the best word segmentation method based on these probability results. The rule-based word segmentation method matches each character or word in the text with the corresponding dictionary one by one. If the match is successful, the character or word is used as a word segmentation result.

[0067] Part-of-speech tagging: Part-of-speech tagging mainly refers to marking the corresponding part of speech for each word segmentation result, such as names of people, places or other parts of speech. Part-of-speech tagging is of great significance in NLP tasks. First, it can eliminate ambiguity, because some words may have different meanings in different contexts or different usages. For example, the word "apple" can mean both fruit and brand. At this time, we can use part-of-speech tagging to distinguish the meaning of the word. Secondly, part-of-speech tagging technology can enhance the characteristics of words and avoid "apple" with two meanings being treated as synonyms, thereby reducing the misleading effect on subsequent analysis.

[0068] In one embodiment, the present application utilizes the part-of-speech tagging function of the Jieba tool, and combines it with a custom place name dictionary to perform corpus preprocessing on the geographic corpus constructed above, to ensure the accuracy of word segmentation of the geographic corpus, while enhancing the geographic information semantics of the corpus and improving the quality of the geographic corpus.

[0069] In one embodiment, the training of the BERT model includes a pre-training stage and a fine-tuning stage. The pre-training stage is used to train unlabeled raw corpus, and the fine-tuning stage is used to adjust model parameters based on preset applications to train labeled corpus.

[0070] Furthermore, the pre-training stage includes masked language model training and next sentence prediction training. Through the masked language model training, the BERT model learns the representation of each geographical word in different contexts, and through the next sentence prediction training, the BERT model learns the relationship between sentences.

[0071] Furthermore, the fine-tuning stage includes sentence pair classification training, single sentence classification training, question-answering training and sequence labeling training.

[0072] In the specific implementation, for step 2, the following contents are included:

[0073] After the construction of the geographic corpus is completed, it is input as a training data sample into the language model, and then the geographic word vector of each geographic word in the corpus can be obtained. This embodiment uses the Bidirectional Encoder Representation from Transformers (BERT) model as a language model to obtain language-environmentalized geographic word vectors, thereby improving the efficiency of geographic information recognition capabilities.

[0074] The overall framework of the BERT model includes two layers: pre-training and fine-tune. The pre-training stage of the BERT model will train a large amount of unlabeled raw data, and then enter the fine-tuning stage to adjust the model parameters based on specific applications to train labeled data.

[0075] Specifically, the BERT model also defines some flags with specific meanings: for example, the [CLS] flag is generally located at the very beginning of the first sentence; the [SEP] flag is generally located in the middle and at the end of the two sentences to be separated; the [MASK] flag is used to predict hidden words, and the BERT model predicts the words hidden by the [MASK] flag by outputting the [MASK] vector.

[0076] The input of the BERT model is the sum of three parts: Token Embedding, Segment Embedding, and Position Embedding. Token Embedding represents the vector of a word; Segment Embedding is used to distinguish which sentence a word belongs to; Position Embedding indicates the position of a word in a sentence (in the BERT model, it is assumed that the maximum length of a sentence is 512). After being processed by the multi-layer Transformer encoder of the BERT model, each output word vector combines the context information of the entire sentence, which enables the output word vector to better represent the meaning of a word in a specific context.

[0077] After receiving the input word vector, the BERT model enters the pre-training stage to train the model. Pre-training mainly includes two training tasks: Masked Language Model (MLM) and Next Sentence Prediction (NSP). In the Masked Language Model task, some words in the input text will be randomly replaced with the [MASK] flag, and then the model predicts these masked words based on the context information. This task helps the model learn the representation of each word in different contexts and enhances the semantic understanding of words. In the next sentence prediction task, the model receives two sentences as input, and then predicts whether they are two adjacent sentences through the encoded information of the [CLS] flag. Through this task, the model can learn the relationship between sentences, so as to better handle natural language processing tasks that require sentence-level understanding.

[0078] After the pre-training stage, the BERT model can enter the fine-tuning stage for specific application tasks. In actual task applications, the BERT model mainly performs the following four NLP tasks: sentence pair classification, single sentence classification, question answering, and sequence labeling.

[0079] In specific implementation, the embodiment of the invention performs place name recognition and extraction technology based on the BiLSTM-CRF model. The main network structure of the BiLSTM-CRF model consists of a bidirectional long-term memory network model (BiLSTM, Bidiractional Long ShortTerm Memory) and a random conditional field model (CRF, conditional random field).

[0080] The bidirectional long short-term memory network model combines forward LSTM and backward LSTM for each unit sequence, and merges their outputs at the same time as the final output. In this way, for each moment, each word sequence has corresponding forward and backward information.

[0081] The conditional random field model is mainly used to calculate and determine the best joint probability of a sequence, which is an optimization at the entire sequence level. In the named entity recognition task, although the BiLSTM model can handle long-interval sentence information well, it is helpless when facing the dependency between close-range labels. The CRF model can obtain the best prediction result from the neighboring labels, which just makes up for the shortcomings of the BiLSTM model. Therefore, the CRF model is used to optimize the results obtained by the BiLSTM model, and finally an optimal label sequence is obtained.

[0082] The score calculation function for any sequence X = (X1, X2, ..., Xn) and its predicted sequence Y = (Y1, Y2, ..., Yn) is as follows:

[0083]

[0084] in is the state transition matrix, representing the transition from label y i Move to label y i+1 The score of P i,yi is a position score matrix, indicating that a given input sequence gets label y at position i i The confidence level, y in the formula i Corresponding to the predicted sequence Y i . Predict the probability of sequence Y:

[0085]

[0086] in, Representing all possible label sequences, the likelihood function of the predicted sequence Y is as follows:

[0087]

[0088] After the CRF model optimizes the results of the BiLSTM model, the output sequence of the maximum score is as follows:

[0089]

[0090] Among them, Y * represents the optimal label sequence, that is, the most likely label sequence predicted for a given input sequence X. Represents the input sequence X and the predicted label sequence score, arg max() means taking the maximum value.

[0091] In one embodiment, the method further comprises:

[0092] The evaluate function of the Kashgari library file is used to evaluate the model quality of the trained BiLSTM-CRF model.

[0093] Specifically, the evaluation indicators of the model quality include precision, recall and F1 measure. The F1 measure is a comprehensive value of precision and recall. The calculation formulas of each evaluation indicator are as follows:

[0094]

[0095]

[0096]

[0097] Among them, P is the precision rate, which represents the proportion of correctly predicted results to the total predicted samples; R is the recovery rate, which represents the proportion of correctly predicted results to all samples; the F1 value combines the two indicators of P and R, and can comprehensively reflect the overall performance of the whole process through one value. a is the number of place names correctly recognized by the BiLSTM-CRF model, A is the total number of place names, and B is the number of place names that have been recognized.

[0098] In order to verify the feasibility and efficiency of the proposed method, the experiment will also compare the annotated corpus of People's Daily, the self-made geographic corpus, and the geographic corpus (the corpus of this paper) based on the combination of People's Daily and the self-made geographic corpus. The results are as follows: Figure 1 As shown in Table 1:

[0099] Table 1 Evaluation index results

[0100] method Precision P Recovery rate R F1 value People's Daily Annotated Corpus 79.2％ 81.50％ 79.63％ Homemade Geographical Corpus 93.10％ 90.49％ 96.30％ This article corpus 97.02％ 96.92％ 96.97％

[0101] The results show that the accuracy of the method of this application (the geographical corpus combined with the People's Daily and the self-made geographical corpus) is 97.02%, the recovery rate reaches 96.92%, and the F1 value reaches 96.97%. It is obvious that the method of this embodiment has the best effect on the recognition and extraction of text geographical names. Analysis shows that: the method based only on the annotated corpus of the People's Daily has a poor recognition effect on some new geographical information because its corpus was generated a long time ago and the amount of geographical information is not large; the latter two methods add the self-made geographical corpus, which makes the geographical information of the corpus rich and fresh, and enhances the semantics of the geographical information of the corpus. The three evaluation indexes have been significantly improved, but the corpus of the method of this embodiment is the largest, and all indicators have achieved the best results.

[0102] The method of the present application is described in detail below with a specific example, including the following steps:

[0103] Step 21, Geographical corpus construction

[0104] Step 21.1, Corpus Selection

[0105] The text-based construction and design working methods of general corpora (including the People's Daily Chinese corpus, MSRA Chinese corpus, etc.) are relatively mature. The corpus mainly studied in this embodiment is a geographic corpus dedicated to the field of geography. The geographic corpus uses geographic entities or text corpora with a relatively large correlation with geographic entities as data sources. Under a unified geographic entity labeling system, geographic information labels are set on the data set through natural language processing tools to form a geographic corpus. Since the current corpus construction results for the geographic field are still relatively scarce, in order to solve this problem, this embodiment will combine the existing mature general corpus-the People's Daily corpus, and in order to fully take into account the representativeness, balance, standardization and scale of the corpus, it is ensured that the geographic information corpus to be studied in this application can truly and directly reflect the role and characteristics of geographic entities in various natural languages, assist computers in identifying the use scenarios of various geographic entities, obtain some geographic information knowledge existing in text and context, and integrate the self-crawled geographic text corpus to enhance its geographic semantics.

[0106] Step 21.2, Geographic Text Corpus

[0107] The corpus constructed in this application is a combination of the 1998 edition of the People's Daily annotated corpus and the self-made geographic text corpus. The 1998 edition of the People's Daily dataset is a corpus jointly developed and created by China Fujitsu Co., Ltd. and the Peking University Computer Research Center. Although the version of the corpus is relatively old, it can still be seen in various competitions and practices. It has always been used as a standard corpus for reference by professionals. There are more than 1.75 million words in the corpus, involving about 28,000 geographical nouns. In this embodiment, the self-made geographic text corpus uses web crawler technology to obtain the "travel" and "food" topic texts on the Sina Weibo website and the 2020 news scripts of the news program. After sorting the crawling results, about 3.1 million words of raw text corpus are obtained, and it is integrated with the annotated corpus of the People's Daily after data preprocessing such as word segmentation and part-of-speech tagging, together as the data set of this topic.

[0108] (1) Constructing Geography Text Corpus

[0109] Regarding crawling texts from Weibo websites, since the two hot topics of "travel" and "food" are more closely related to geographic information than other topics, and crawling Weibo under these topics does not require user login, the crawling method used is relatively easy. Therefore, this embodiment chooses to crawl the text content of Sina Weibo under the two hot topics of "travel" and "food" as one of the sources of the self-made geographic corpus of this embodiment.

[0110] To avoid being interfered by the anti-crawling mechanism of the Weibo website, you first need to set the HTTP request header (mainly set the three parameters of Host, Requester, and User-Agent) to simulate a browser accessing the Weibo website;

[0111] Use the GET request method to crawl the microblogs with the topic of "travel" or "food" by page number (only the first 200 pages are crawled in this embodiment). In order to more conveniently construct a URL with parameters, the params parameter is used in the experiment;

[0112] However, if it is a long microblog, you must click on the "Expand full text" link to browse and read the complete microblog text. Analyzing its page structure, it is found that the "Expand full text" link is composed of "https: / / m.weibo.cn / status / "+id, where id is the id of the corresponding microblog article. Therefore, based on the above code, it is also necessary to determine whether it is a long microblog. If so, it is necessary to jump to the corresponding page according to its id and crawl again;

[0113] Observe the relevant fields and find that there is a Boolean field named "isLongText". Therefore, when parsing the json data returned by the page, judge whether it is a long text based on the value of this field. If it is not a long text, directly extract the text field content. If it is a long text, you need to call the getLongText function defined above to get the complete long text text field content.

[0114] Finally, in order to facilitate the organization of the result data, all the crawled results are stored in a table with two columns. The first column is the time when the Weibo was published, and the second column is the complete content of the Weibo text.

[0115] However, during the code running process, it was found that the local IP would be blocked and crawling would stop because the data crawling interval was too fast or the number of crawling times was too many. To solve this problem, this embodiment adopts the method of building a proxy IP pool, that is, when a certain IP is blocked, a proxy IP is randomly provided in the constructed IP pool and its availability is tested. Then, the original IP address is used to connect to the network IP address and continue crawling, thereby solving the problem of an IP being blocked due to frequent requests. The source of the proxy IP pool constructed this time is the free IP provided by the "Fast Proxy" website.

[0116] Since the language of Weibo text is quite arbitrary, and news programs are nationally certified news portals with advantages such as authoritative resources, standardized terms, and wide coverage, this embodiment will use the 2020 news program press releases as the expanded corpus, which can strengthen the standard authority of the corpus language on the one hand, and enhance the geographical semantics of the corpus on the other.

[0117] By analyzing the official website of the news program, it is found that after selecting a date on the calendar control, a list of news for that day will appear below. The first item in the list is the full news program of the day, followed by each specific news item. Figure 2 shown.

[0118] Open F12 debugging, then click on different dates to find previous requests in the XHR tab. The only thing that changes is the date string in the link address, such as Figure 3 As shown by Figure 3 The crawling ideas can be determined:

[0119] ①Get the news list of the day according to the changed date;

[0120] ② Save the content of news manuscripts in a loop.

[0121] This crawler needs to crawl news for the whole year of 2020. It is necessary to use the pandas library to generate a list of 365 dates from 20200101 to 20201231, so that these dates can be used as parameters to call functions in the for loop.

[0122] Through analysis, it is found that the links of each news item in the HTML code of the news program webpage are Tags The href attribute content of the tag, so the GET request method is used to obtain the content under the tag, that is, the links of the news programs on a certain date. After obtaining the news link, the find method of the BeautifulSoup library is still used in the GET request method to obtain <div class＝"cnt_bd”> The text content under the first tag after the tag is the complete news text content corresponding to the link, and is stored in a txt document named after the date. At this point, the work of self-made geography text corpus has been completed. The crawling result format is as follows Figure 3 As shown, this crawl obtained a total of about 10,000 texts, totaling more than 3.1 million words.

[0123] (2) Constructing a place name dictionary

[0124] Since the sample data of the geographic word vector training model needs to have geographic information tags, before training the geographic word vector, the geographic corpus needs to be labeled with place name tags word by word. The present invention uses the "word segmentation" and "part-of-speech tagging" functions of the Jieba word segmentation tool, and adds a custom place name address dictionary to match the place name information in the geographic text corpus, which is convenient for subsequent labeling of geographic information tags.

[0125] The self-defined place name address dictionary of the present invention consists of two parts: the place names of the five administrative regions in China from the website of "Yigecun" and the names of all popular scenic spots from "Qunar.com". The geographical data stored in the website of "Yigecun" is relatively authoritative, and its database mainly covers provinces, cities, counties (districts), towns (streets), and villages across the country, including 34 provincial administrative regions, 340 municipal administrative regions, 2844 county administrative regions, 19883 town administrative regions, and 38773 village administrative regions.

[0126] The general idea of ​​crawling the five-level administrative districts of the "Yigecun" website is as follows. The crawling results are as follows Figure 5 As shown,

[0127] ①Request to obtain the HTML page;

[0128] ②Convert HTML into BeautifulSoup;

[0129] ③Get the required data according to the grammatical rules of BeautifulSoup;

[0130] ④Save the data into a document.

[0131] However, people often go to specific attractions located in these administrative regions when traveling. That is, the destination place names are not the names of the administrative regions. Therefore, all the popular attraction names of the most popular travel website "Qunar.com" are used as the expansion of the place name and address dictionary. The website contains about 660,000 popular attractions across the country, and the attraction resources are very rich and complete.

[0132] Similarly, for crawling attractions on Qunar.com, a recursive custom function is still used to obtain all attraction data based on tags.

[0133] Finally, we need to define the part of speech for each word in the dictionary. Since each word in the dictionary represents a place name, we uniformly define the part of speech as "LOC". The final dictionary format is as follows: Figure 6 The custom place name dictionary has been completed, which includes more than 720,000 place names. The large scale has greatly improved the accuracy of Jieba's "part-of-speech tagging" of geographic text corpus.

[0134] Step 21.3, Corpus Preprocessing

[0135] The steps of corpus preprocessing mainly include cleaning, word segmentation, part-of-speech tagging, etc.

[0136] In the field of natural language processing, the geographical information in the geographical corpus is labeled with corresponding labels in combination with Jieba tools and custom dictionaries. This is called sequence labeling. Common labeling systems include BIOE system, BIO system and BIOES system. This embodiment uses the BIO labeling system to process and produce training sample data. The labeling system has three labels: O, B-LOC and I-LOC, where O represents non-geographic information, B-LOC represents the first Chinese character of geographic information, and I-LOC represents the remaining content after the first Chinese character of geographic information. The labeled sample data is as follows: Figure 7 shown.

[0137] First, Jieba segmentation is performed on the geographic text corpus that needs segmentation, and the custom place name dictionary prepared in advance is loaded at the same time. After Jieba completes the segmentation according to the dictionary, each segmentation result will refer to the custom place name dictionary. If the segmentation result appears in the custom dictionary, its part of speech is the part of speech marked by the corresponding item in the dictionary, and the position of each Chinese character in the segmentation result is determined (first character or non-first character). If it does not appear in the dictionary, it is marked as "O". Finally, it can be written into a txt document.

[0138] At this point, the data set of this embodiment has been processed completely. Finally, the result data needs to be randomly divided into three parts in the ratio of 1:7:2, which are used as the verification set, training set and test set of the BERT model and BiLSTM-CRF model respectively. The verification set data is mainly used to debug the model, and the training set is mainly used to learn the model. After finding the best parameters on the training set, it is tested once on the test set data. The error result of the test set data should be used as the final error approximation.

[0139] Step 22: Build BERT geographic word vector

[0140] After the construction of the geographic corpus is completed, it is input into the language model as a training data sample, and then the geographic word vector of each geographic word in the corpus can be obtained. The present invention uses the BERT model to obtain the language-environmentalized geographic word vector, thereby improving the efficiency of geographic information recognition capabilities. The overall framework of BERT includes two layers: pre-train and fine-tune. The pre-training stage of the BERT model will train a large amount of unlabeled raw corpus, and then enter the fine-tuning stage to adjust the model parameters based on specific applications to train the labeled corpus.

[0141] In addition, the BERT model also defines some flags with specific meanings: for example, the [CLS] flag is generally located at the very beginning of the first sentence; the [SEP] flag is generally located in the middle and at the end of the two sentences to be separated; the [MASK] flag is used to predict hidden words, and the BERT model predicts the words hidden by the [MASK] flag by outputting the [MASK] vector.

[0142] The input of the BERT model is the sum of three parts: Token Embedding, Segment Embedding, and Position Embedding. Token Embedding represents the vector of a word; Segment Embedding is used to distinguish which sentence a word belongs to; Position Embedding indicates the position of a word in a sentence (in the BERT model, it is assumed that the maximum length of a sentence is 512).

[0143] After receiving the input word vector, the BERT model enters the pre-training stage to train the model. Pre-training mainly includes two training tasks. The first training task is to predict replacement words, that is, to randomly replace some words in the sentence with the [MASK] flag, and then input the sentence into the BERT model for encoding, and finally predict the correct result of the replacement position by using the encoding information of the [MASK] flag. The second training task is to predict the previous and next sentences, which mainly involves inputting two sentences into the BERT model separately, and then predicting whether they are two adjacent sentences by using the encoding information of the [CLS] flag.

[0144] After the pre-training stage, the BERT model can enter the fine-tuning stage for specific application tasks. In actual task applications, the BERT model mainly performs the following four NLP tasks: sentence pair classification, single sentence classification, question answering, and sequence labeling.

[0145] Since the BERT model is a pre-trained learning model, that is, based on the pre-trained model, a targeted fine-tuning layer network structure is added, and a relatively small amount of corpus is used to train the fine-tuning layer so that it can meet certain task requirements, the experiment in this embodiment needs to download the trained BERT model in advance. The present invention downloads a Chinese BERT model disclosed by Google. The downloaded file path should be set to the value of the BERT_PATH parameter in the BERTEmbedding function so that the function can correctly load the required file. The sequence_length parameter only accepts 'int' values, so it is set to 100, that is, the input and output sequence lengths will be set to 100. The key code is shown in the figure:

[0146] Step 23: Model training

[0147] This embodiment is based on the place name recognition and extraction technology of the BiLSTM-CRF model. The main network structure of the BiLSTM-CRF model consists of a bidirectional long-term memory network model (Bi-LSTM) and a random conditional field model (CRF).

[0148] BiLSTM (Bidiractional Long Short Term Memory) combines forward LSTM and backward LSTM for each unit sequence, and merges their outputs at the same time as the final output. In this way, for each moment, each word sequence has corresponding forward and backward information.

[0149] The conditional random field (CRF) model is mainly used to calculate and determine the best joint probability of a sequence, which is an optimization at the entire sequence level. In the named entity recognition task, although the BiLSTM model can handle long-interval sentence information well, it is helpless when facing the dependency between close-range labels. The CRF model can obtain the best prediction result from the neighboring labels, which just makes up for the shortcomings of the BiLSTM model. Therefore, this lesson uses the CRF model to optimize the results obtained by the BiLSTM model, and finally obtains an optimal label sequence.

[0150] The score calculation function for any sequence X = (X1, X2, ..., Xn) and its predicted sequence Y = (Y1, Y2, ..., Yn) is as follows:

[0151]

[0152] Among them, P is the result score matrix of the BiLSTM model, P i,j refers to the score of the jth label of the i-th word. A is the change matrix, A i,j It represents the score after label i changes to label j. The probability of predicting sequence Y is as follows:

[0153]

[0154] The likelihood function for predicting the sequence Y is as follows:

[0155]

[0156] After the CRF model optimizes the results of the BiLSTM model, the output sequence of the maximum score is as follows:

[0157]

[0158] During the training process, the geographic word vector that has been fine-tuned by the BERT model is used as the input vector, and deep learning based on the BiLSTM-CRF neural network is performed to obtain the final model for automatic recognition of text place name entities and save it. The number of training rounds (epochs) in the BiLSTM-CRF model is set to: epochs = 20, and the number of data samples (batch_size) sent to the model for learning each time training is set to: batch_size = 512.

[0159] The embodiments provided by the present invention improve the overall performance of geographic information recognition and positioning by constructing word vectors with strong geographic semantics, thereby solving the shortcomings of the prior art in terms of inaccurate semantic understanding in geographic information processing. The results show that the geographic corpus obtained by collating the general geographic corpus with the self-made corpus of this application has achieved the best results in terms of precision, recovery rate and F1 value. The precision of this application is 8 percentage points higher than that of the general geographic corpus and 4% higher than that of the separate self-made corpus. Therefore, this application can extract all relevant geographic names more accurately.

[0160] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing geographic word vectors for user positioning, characterized in that: include: Use geographic entities and text corpora associated with them as data sources to build a geographic corpus; Input the geographic corpus into the BERT model for training to obtain a geographic word vector for each geographic word in the geographic corpus, wherein the geographic word vector includes context information of the geographic word in the sentence; The BiLSTM-CRF model is trained based on the geographic word vector, and the trained BiLSTM-CRF model is used to recognize and extract place names.

2. The method for constructing a user-oriented geographic word vector according to claim 1, characterized in that: The method of using geographic entities and text corpora associated with the geographic entities as data sources to construct a geographic corpus includes: Use web crawler technology to obtain texts and news transcripts on tourism and food topics from websites, and then organize them to obtain raw text; Perform data preprocessing on raw corpus text; The preprocessed raw corpus text is integrated with the People's Daily annotated corpus to obtain the geographic corpus.

3. The method for constructing a user-oriented geographic word vector according to claim 1, characterized in that: The method further comprises: The geographic corpus is subjected to corpus preprocessing, wherein the corpus preprocessing includes cleaning, word segmentation and part-of-speech tagging.

4. The method for constructing a user-oriented geographic word vector according to claim 3, characterized in that: The cleaning methods include an arrangement method, a manual deduplication method, a marking method and a deletion method.

5. The method for constructing a user-oriented geographic word vector according to claim 3, characterized in that: The part-of-speech tagging adopts Jieba tool.

6. The method for constructing a user-oriented geographic word vector according to claim 1, characterized in that: The training of the BERT model includes a pre-training stage and a fine-tuning stage. The pre-training stage is used to train unlabeled raw corpus, and the fine-tuning stage is used to adjust model parameters based on preset applications to train labeled corpus.

7. The method for constructing a user-oriented geographic word vector according to claim 6, characterized in that: The pre-training stage includes masked language model training and next sentence prediction training. Through the masked language model training, the BERT model learns the representation of each geographic word in different contexts. Through the next sentence prediction training, the BERT model learns the relationship between sentences.

8. The method for constructing a user-oriented geographic word vector according to claim 6, characterized in that: The fine-tuning stage includes sentence pair classification training, single sentence classification training, question answering training and sequence labeling training.

9. The method for constructing a user-oriented geographic word vector according to claim 1, characterized in that: The method further comprises: The evaluate function of the Kashgari library file is used to evaluate the model quality of the trained BiLSTM-CRF model.

10. The method for constructing a user-oriented geographic word vector according to claim 9, characterized in that: The evaluation indicators of the model quality include precision, recovery rate and F1 value. The F1 value is the comprehensive value of precision and recovery rate. The calculation formula of each evaluation indicator is: Among them, P is the precision rate, R is the recovery rate, a is the number of place names correctly recognized by the BiLSTM-CRF model, A is the total number of place names, and B is the number of place names that have been recognized.

Images