An automatic disaster event extraction method for public information sources

Abstract

The present invention provides a method for automatically extracting disaster events for public information sources, including: using a hybrid disaster crawler to crawl disaster-related web pages on the Internet, and realizing crawl optimization of disaster-themed pages based on a crawling depth adaptive control strategy; web page preprocessing, identifying the tag type of the crawled web pages, and extracting event description texts; event entity extraction, performing word segmentation processing on the extracted event description texts, and constructing event entity feature vectors by identifying and parsing parts of speech, pause words, and entity keywords; classifying text types according to the constructed event entity feature vectors, and generating disaster information maps according to the corresponding relationship between disaster events and description attributes. The present invention can use public Internet web page resources to mine information on disaster events, effectively expanding the information acquisition channels and intelligent acquisition levels of disaster events, and has important value for establishing disaster databases and researching disaster early warning technologies.

Description

Technical Field

[0001] The invention relates to the field of network information collection, and in particular to an automatic disaster event extraction method oriented to public information sources. Background Art

[0002] Natural disasters have typical characteristics such as many types, wide distribution, high frequency and heavy losses. With the global climate change and the continuous acceleration of China's economic development and urbanization, the situation of natural disaster prevention in my country is becoming increasingly severe. In order to strengthen the scientific and technological support for disaster prevention and enhance the application of digital technology, government departments at all levels, research institutes and technology companies have carried out technical analysis and system development related to disaster monitoring.

[0003] At present, the relevant information of disaster events mainly depends on the reporting and manual integration of departments at all levels. Considering the security protection needs of detailed information, it is difficult for the technical research and development department to obtain the original information report. Therefore, the existing monitoring system usually provides services in the form of open interfaces, and the specific event information mainly relies on manual entry. At the same time, in order to strengthen the early warning and judgment capabilities of disaster events, it is often necessary to jointly analyze the specific circumstances of disaster events with other factors in the corresponding time and space, such as meteorology, policies, and behaviors. The difficulty in obtaining event information will lead to slow progress in such research.

[0004] Currently, web pages have become an important carrier for public information release. Massive amounts of news, announcements, comments, etc. are uploaded and released on the Internet at all times, and most of them are publicly accessible. Considering the huge amount of information on the Internet, web pages contain a large amount of disaster event information that can be safely obtained. Therefore, actively discovering disaster events from public information sources on the Internet is of great value in strengthening disaster monitoring capabilities and improving disaster assessment levels. Summary of the invention

[0005] The present invention provides a method for extracting disaster events from public text information on web pages. The technical problem it solves is the difficulty in obtaining events in natural disaster monitoring systems and related technical research. A hybrid disaster crawler is designed to adaptively capture disaster-related web pages through network crawler technology, and to process web page content through web page parsing and semantic parsing technology to extract key information elements of disaster events in web page texts.

[0006] The specific technical solutions of the present invention are as follows:

[0007] A method for extracting disaster events from public text information on web pages, comprising the following steps:

[0008] Step 1: Use the hybrid disaster crawler to crawl disaster-related web pages on the Internet, and optimize the crawling of disaster-themed pages based on the crawling depth adaptive control strategy;

[0009] Step 2: Web page preprocessing: identify the tag type of the web pages crawled in step 1 and extract the event description text;

[0010] Step 3: Event entity extraction: perform word segmentation on the event description text extracted in step 2, and construct an event entity feature vector by identifying and parsing parts of speech, pause words, and entity keywords;

[0011] Step 4: Classify the text type according to the event entity feature vector constructed in step 3, and generate a disaster information map based on the correspondence between disaster events and description attributes.

[0012] Furthermore, in step 1, the hybrid disaster crawler is used to crawl disaster-related web pages on the Internet, specifically including:

[0013] The search engine's retrieval results are used as seed pages to determine whether the pages in the result list already exist in the disaster theme site directory page library. For pages that do not exist in the directory page library, the crawling depth adaptive control strategy is used to crawl related links.

[0014] Among them, the crawling depth adaptive control strategy is specifically as follows: for emergency theme site pages, the page is crawled by using the site traversal to find similar event notification pages; for ordinary news web pages, K-layer depth is selected for crawling, and the crawled K-layer link addresses are stored in the list to be analyzed for further analysis; and irrelevant pages are directly filtered.

[0015] Furthermore, the web page preprocessing process of step 2 includes: extracting the web page source code of the web page to be analyzed crawled in step 1, judging the tag category, extracting the character strings contained in all text class tags and storing them locally, and marking the tag attributes corresponding to the character strings for subsequent event entity extraction and feature vector construction needs.

[0016] Furthermore, the event description text in step 2 is converted into a set of event entity feature vectors in step 3, where each event entity feature vector corresponds to a descriptive entity phrase. Specifically, the model of the event entity feature vector is a six-tuple, recorded as (Fre, Pos, W, Pro, Ten, Sg), and each element represents the frequency, position, weight, entity attribute, tense and spatial description granularity of the corresponding phrase in turn;

[0017] In step 3, the event entity feature vector is constructed, which specifically includes:

[0018] First, a set of descriptive entity phrases are extracted from the event description text in step 2. The specific process includes: performing string segmentation on the event description text, and semantically parsing the segmented words based on part of speech, pause word annotation, disaster theme vocabulary, and string position, and merging related phrases to enhance the accuracy of the event description. The total number of merged phrases is recorded as N. T ;

[0019] Next, the phrases are classified by attributes of "time (T)", "location (L)", "state (S)" and "event type (E)", and the semantic similarity analysis is performed on the descriptive entity phrases under each attribute. Similar phrases are merged again to obtain descriptive entity phrases. Among them, t, l, r, and e represent the number of phrases retained in the four types of entity keywords after merging;

[0020] Then, a feature vector is constructed for each phrase based on the event entity feature vector model to describe it. Assume that the phrase since The event entity feature vector of the phrase is recorded as The calculation method of the feature items of each dimension is as follows:

[0021] Frequency of words The calculation formula is:

[0022]

[0023] Among them, n i,j express The frequency of the word in the text of the web page;

[0024] The position element in the feature vector only represents the position of the corresponding phrase when it first appears in the text, and is recorded as follows after normalization:

[0025]

[0026] where s i ,e i Respectively The starting and ending positions of the webpage, Len represents the total length of the string in the webpage;

[0027] The weight is mainly calculated based on the tag level. Let N be the total number of text tags in the event description document. tp level attributes, and the weights of the 1st, 2nd, ..., Nth level labels are w1, w2, ..., w N , arranged in descending order, for phrases For example, let its label level be k, then the corresponding label weight is:

[0028]

[0029] in, Take the maximum value of the weight in the merged subset phrase as its weight, that is The entity attribute is P, which includes four categories: T, L, S, and E.

[0030]

[0031] The tense expression is relative to the time when an event occurs. For time entities, they are divided into three categories: past, present, and future. The tenses of other entities are recorded as empty, as follows:

[0032]

[0033] For place-related phrases that appear in the document, their description granularity is differentiated according to the national administrative divisions, including five levels: province, prefecture, county, township, and village. The spatial granularity of other phrases is recorded as empty, as follows:

[0034]

[0035] Furthermore, in step 4, the text type is classified according to the event entity feature vector constructed in step 3, and a disaster information map is generated according to the correspondence between the disaster event and the description attribute, which specifically includes:

[0036] Get the event entity feature vector constructed in step 3, and use decision tree or naive Bayes classification algorithm to classify the documents, including statistics, progress report, danger warning and cause analysis. Statistics web pages are not included in the analysis when disaster events are extracted. Based on the semantic expression rules of the entity relationship of "cause", "status", "occurrence time", "duration" and "event location", the phrases merged by the segmentation and association word groups in step 3 are parsed to generate a disaster information map.

[0037] Furthermore, the phrases formed by combining the word segmentation and the associated word group in step 3 are parsed. The specific parsing process includes: entity type identification, entity relevance judgment, and associated relationship identification, wherein:

[0038] Entity type recognition refers to judging whether a character string is a description of a disaster event and what attribute description dimension of the disaster event it belongs to, including "place name", "time" and "event type", by judging the character string position, part of speech and the matching degree with keywords in the disaster theme word library.

[0039] Entity relevance judgment is to take all the phrases describing the event type of disaster events in the entity type recognition results as the center, compare the position distances of other descriptive entity phrases and the description phrases of each event type, judge the event object described by the descriptive entity phrase, and establish the association relationship between entity phrases;

[0040] Relationship identification is to determine the relationship between various related entity phrases, including "event type", "cause" and "status", based on the attribute description dimensions of pause words and phrases.

[0041] The advantages of the present invention are:

[0042] (1) Using public network information sources to obtain disaster event information is an autonomous and controllable method for obtaining disaster information. At the same time, the disaster descriptions from different information publishers can complement and proofread each other, restoring the disaster scene from multiple angles;

[0043] (2) A hybrid disaster crawler technology is proposed. The disaster-related results given by Internet search engines are used as seed pages. By analyzing the attributes of the source sites of the seed pages, the crawler's crawling depth of related pages is adaptively adjusted. This effectively solves the problem of missing a large number of disaster-related web pages when the disaster theme vocabulary is incomplete, while avoiding the waste of time and computing resources caused by deep crawling.

[0044] (3) The page publishing attributes of disaster events can be predicted through the event entity feature vector, and the event information pages that need to be analyzed and extracted can be screened again. Web pages used for disaster statistical analysis, publicity and education, etc. can be filtered out from the pages to be analyzed that contain disaster keywords. Only pages with relatively complete description dimensions of disaster events can be mined, which is conducive to improving the efficiency of event information extraction and the accuracy of relationship recognition of event entities. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of one embodiment of a method for automatically extracting disaster events from public information sources according to the present invention;

[0046] Figure 2 It is the technical roadmap of the hybrid disaster crawler in the present invention;

[0047] Figure 3 is a block diagram of information extraction of disaster events in the present invention;

[0048] Figure 4 It is a schematic diagram of an application example of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The sources of web pages for disaster events include official websites, personal blogs, mainstream media, etc. Different types of web pages have different characteristics for event releases. For example, official websites have relatively fixed styles and are responsible for accident notifications. They often provide regular updates on the progress of relevant disaster situations. However, personal websites lack regularity in releases, but sometimes they can provide first-hand on-site information with better real-time information. Therefore, according to the information sources of different websites, the page crawling mechanism needs to be adjusted to improve the information discovery efficiency of web crawlers.

[0051] A complete event element should include multiple knowledge dimensions such as time, location, progress, and cause. However, as the event progresses and the purpose of the announcement varies, the event description contained in a single web page usually has different emphases, language expressions, and information granularity. Therefore, it is necessary to propose a universal disaster event extraction method that can effectively adapt to the extraction of event elements in different announcement types.

[0052] The disaster event extraction method of the present invention will be based on Figure 1 The steps shown are implemented by first using a web crawler to obtain web page resources related to disaster events, pre-processing the pages based on web page parsing technology, and extracting event description text; extracting event entities through semantic parsing technology, constructing entity feature vectors, and implementing text classification; and constructing an information map of disaster events based on the text classification results. In general, the main implementation method of the present invention includes the following steps:

[0053] Step 1: Use the hybrid disaster crawler to crawl disaster-related web pages on the Internet, and optimize the crawling of disaster-themed pages based on the crawling depth adaptive control strategy;

[0054] Step 2: Web page preprocessing: identify the tag type of the web pages crawled in step 1 and extract the event description text;

[0055] Step 3: Event entity extraction: perform word segmentation on the event description text extracted in step 2, and construct an event entity feature vector by identifying and parsing parts of speech, pause words, and entity keywords;

[0056] Step 4: Classify the text type according to the event entity feature vector constructed in step 3, and generate a disaster information map based on the correspondence between disaster events and description attributes.

[0057] Among them, in step 1, a hybrid disaster crawler is used to crawl disaster-related web pages on the Internet, specifically including:

[0058] The search results of Internet search engines are used as seed pages, such as Figure 2 As shown, the disaster crawler will periodically crawl two types of pages. One is to use the search engine to periodically search for disaster-related pages based on the disaster theme vocabulary; the other is to periodically crawl the directory pages in the disaster theme site library and capture the link pages updated by the directory pages for analysis.

[0059] Then, the existence of the relevant web pages in the search engine result list is first judged. If such pages already exist in the subject site library, they are directly sent to the web page parser for event information extraction. Otherwise, they are stored in the list to be analyzed. In addition to the web page parsing process, the crawling depth adaptive control strategy is also required to crawl related links. Through site attribute recognition, disaster-related pages can be divided into three categories: emergency subject site pages, ordinary news pages, and irrelevant pages.

[0060] The crawling depth adaptive control strategy is specifically as follows:

[0061] For the disaster information page under the emergency theme site, other disaster information that may be released by the site will be mined through the site traversal method, and the site's directory page will be automatically identified and the link address will be stored in the disaster theme site library. For ordinary news web pages, K-layer deep crawling is selected, and the crawled K-layer link addresses are stored in the list to be analyzed for further analysis. The minimum value of K can be 1, or it can be set according to the experience value, or it can be dynamically adjusted according to the number of crawled links. For other irrelevant pages, it means that although the page contains some disaster keywords, it does not contain complete disaster event record information, so it can be filtered directly.

[0062] The web page preprocessing process of step 2 includes: extracting the web page source code of the web page to be analyzed crawled in step 1, judging the tag category, extracting the character strings contained in all text class tags and storing them locally, and marking the tag attributes corresponding to the strings for subsequent event entity extraction and feature vector construction needs.

[0063] In step 3, the event entity feature vector is constructed, which specifically includes:

[0064] The complete event description text will first be processed by word segmentation, and semantic analysis will be performed on the word segmentation through information such as part of speech, pause words, disaster theme, and string position, and related phrases in the context will be merged to enhance the accuracy of the event description. Figure 4 In the disaster announcement text in , after word segmentation, two place nouns "Shenyang" and "Qipanshan" will be obtained. Since the two are related to each other and both modify the disaster phenomenon of "wildfire", they can be combined and recorded as "Shenyang Qipanshan". The total number of combined phrases in the present invention is recorded as N T .

[0065] Next, the merged phrases are attributed and merged again to construct event entity feature vectors. Figure 3 As shown in the figure, the phrases after word segmentation and merging will be classified according to the attributes of "time (T)", "place (L)", "state (S)" and "event type (E)". By performing semantic similarity analysis on the descriptive entity phrases under various attributes, similar phrases are merged again to obtain the descriptive entity phrases of the text. Among them, t, l, r, and e represent the number of phrases retained after the merging process for the four types of entity keywords. Figure 4 Taking the progress report of the Qipanshan wildfire in Shenyang as an example, "2019-04-19", "April 19th" and "19th" can be combined into "April 19, 2019".

[0066] The event entity feature vector proposed in this invention is a six-tuple, recorded as (Fre, Pos, W, Pro, Ten, Sg). Next, a six-tuple feature vector will be established for each descriptive entity phrase, and each element represents the frequency, position, weight, entity attribute, tense and spatial description granularity of the corresponding phrase. since The calculation method of each feature dimension of the phrase is as follows.

[0067] Frequency of words The calculation formula is

[0068]

[0069] Among them, n i,j express The frequency of words appearing in the text of the web page. Since a phrase may appear multiple times, the position element in the feature vector only represents the position of the corresponding phrase when it first appears in the text. After normalization, it can be recorded as follows

[0070]

[0071] s i ,e i Respectively The starting and ending positions of the web page, Len represents the total length of the string in the web page.

[0072] The weight is mainly calculated based on the tag level. Let N be the total number of text tags in the event description document. tp level attributes, and the weights of the 1st, 2nd, ..., Nth level labels are w1, w2, ..., w N , arranged in descending order. For example, let its label level be k, then the corresponding label weight is

[0073]

[0074] in, Take the maximum value of the weight in the merged subset phrase as its weight, that is The entity attribute is P, which includes four categories: T, L, S, and E.

[0075]

[0076] Tense is expressed relative to the time when an event occurs. Therefore, time entities can be divided into three categories: past, present, and future. The tense of other entities is recorded as empty.

[0077]

[0078] For place-related phrases appearing in the document, their description granularity can be differentiated according to the national administrative divisions, including five levels: province, prefecture, county, township, and village. The spatial granularity of other phrases is recorded as empty.

[0079]

[0080] In step 4, the text type is classified according to the event entity feature vector constructed in step 3, and a disaster information map is generated according to the correspondence between the disaster event and the description attribute, which specifically includes:

[0081] Based on the event entity feature vector constructed in step 3, the text attributes of disaster events can be classified into statistics, progress report, danger warning and cause analysis through decision tree, naive Bayes and other classification algorithms. Figure 3 As shown, the statistical pages will be filtered directly because the information they contain is too mixed, and the event description granularity is usually large, so the information extraction accuracy is not high. Figure 4 The example shown is a progress report text, so the entity relationships usually include "status", "occurrence time", "release time", "duration", "event location", etc. According to the semantic expression rules of entity relationships, the phrases after the segmentation and association word groups in the above steps can be parsed to generate a disaster information map.

[0082] Taking the "state" entity relationship as an example, the ternary graph mapping (h, r, t) often has the following characteristics: the front entity h and the tail entity t represent the event type and event state respectively, and the entity relationship r represents the state evolution action, such as "has", "becomes", etc. Therefore, the description sentence "The Qipan Mountain wildfire has been completely extinguished" in the page can be parsed according to the following steps: 1) Entity type identification: ("Qipan Mountain", place name), ("wildfire", event type), ("has", state pause word), ("all", state pause word), ("extinguish", state verb); 2) Entity relevance judgment: There are two pairs of relevance entities, "Qipan Mountain"-"wildfire" and "wildfire"-"extinguish"; 3) Relevance identification: From the place name attribute of "Qipan Mountain", it can be known that it represents the "location" of "wildfire", and according to the semantic expression rules of the "state" entity relationship, it can be known that the current state of "wildfire" is "extinguished".

[0083] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A disaster event extraction method for web page public text information, characterized in that The following steps are involved: Step 1: Use the hybrid disaster crawler to crawl disaster-related web pages on the Internet, and optimize the crawling of disaster-themed pages based on the crawling depth adaptive control strategy; Step 2: Web page preprocessing: identify the tag type of the web pages crawled in step 1 and extract the event description text; Step 3: Event entity extraction: perform word segmentation on the event description text extracted in step 2, and construct an event entity feature vector by identifying and parsing parts of speech, pause words, and entity keywords; Step 4: Classify the text type according to the event entity feature vector constructed in step 3, and generate a disaster information map based on the correspondence between disaster events and description attributes; The event description text in step 2 is converted into a set of event entity feature vectors in step 3, where each event entity feature vector corresponds to a descriptive entity phrase. Specifically, the model of the event entity feature vector is a six-tuple, denoted as , each element represents the frequency, position, weight, entity attribute, tense and spatial description granularity of the corresponding phrase in turn; In step 3, the event entity feature vector is constructed, which specifically includes: First, a set of descriptive entity phrases are extracted from the event description text in step 2. The specific process includes: performing string segmentation on the event description text, and semantically parsing the segmented words based on part of speech, pause word annotation, disaster theme vocabulary, and string position, merging related phrases to enhance the accuracy of event description, and the total number of merged phrases is recorded as ; Next, the phrases are classified by attributes such as "time (T)", "location (L)", "state (S)" and "event type (E)", and the semantic similarity analysis is performed on the descriptive entity phrases under each attribute. Similar phrases are merged again to obtain descriptive entity phrases. ,in They represent the number of retained descriptive entity phrases of the four types of attributes after merging; Then, a feature vector is constructed for each phrase based on the event entity feature vector model to describe it. Assume that the phrase since The event entity feature vector of the phrase is recorded as , the calculation method of each dimension feature item is as follows: Frequency of words The calculation formula is: ； in, express The frequency of the word in the text of the web page; The position element in the feature vector only represents the position of the corresponding phrase when it first appears in the text, and is recorded as follows after normalization: ； in , Respectively The starting and ending positions, Represents the total length of the string in the web page; The weight is mainly calculated based on the tag level, and the total number of text tags in the event description document is level attributes, and the weights of the 1st, 2nd, ..., Nth level labels are ,..., , arranged in descending order, for phrases For example, let its label level be , then the corresponding label weight is: ； in, Take the maximum value of the weight in the merged subset phrase as its weight, that is , The entity attribute is P, which includes four categories: T, L, S, and E. ； The tense expression is relative to the time when an event occurs. For time entities, they are divided into three categories: past, present, and future. The tenses of other entities are recorded as empty, as follows: ； For place-related phrases that appear in the document, their description granularity is differentiated according to the national administrative divisions, including five levels: province, prefecture, county, township, and village. The spatial granularity of other phrases is recorded as empty, as follows: 。 2. The method for extracting disaster events from web page public text information as claimed in claim 1, characterized in that: In step 1, the hybrid disaster crawler is used to crawl disaster-related web pages on the Internet, including: The search engine's retrieval results are used as seed pages to determine whether the pages in the result list already exist in the disaster theme site directory page library. For pages that do not exist in the directory page library, the crawling depth adaptive control strategy is used to crawl related links. Among them, the crawling depth adaptive control strategy is specifically as follows: for emergency theme site pages, the page is crawled by using the site traversal to find the event notification page; for ordinary news web pages, the K-layer depth is selected for crawling, and the crawled K-layer link address is stored in the list to be analyzed for further analysis; and irrelevant pages are directly filtered.

3. The method for extracting disaster events from web page public text information as claimed in claim 1, characterized in that: The web page preprocessing process of step 2 includes: extracting the web page source code of the web page to be analyzed crawled in step 1, judging the tag category, extracting the character strings contained in all text class tags and storing them locally, and marking the tag attributes corresponding to the character strings for subsequent event entity extraction and feature vector construction needs.

4. The method for extracting disaster events from web page public text information as claimed in claim 1, characterized in that: In step 4, the text type is classified according to the event entity feature vector constructed in step 3, and a disaster information map is generated according to the correspondence between the disaster event and the description attribute, which specifically includes: Get the event entity feature vector constructed in step 3, and use decision tree or naive Bayes classification algorithm to classify the documents, including statistics, progress report, danger warning and cause analysis. Statistics web pages are not included in the analysis when disaster events are extracted. Based on the semantic expression rules of the entity relationship of "cause", "status", "occurrence time", "duration" and "event location", the phrases merged by the segmentation and association word groups in step 3 are parsed to generate a disaster information map.

5. The method for extracting disaster events from web page public text information as claimed in claim 4, characterized in that: Parse the phrases formed by combining the segmented words and associated word groups in step 3. The specific parsing process includes: entity type identification, entity relevance judgment, and association relationship identification, among which: Entity type recognition refers to judging whether a character string is a description of a disaster event and what attribute description dimension of the disaster event it belongs to, including "place name", "time" and "event type", by judging the character string position, part of speech and the matching degree with keywords in the disaster theme word library. Entity relevance judgment is to take all the phrases describing the event type of disaster events in the entity type recognition results as the center, compare the position distances of other descriptive entity phrases and the description phrases of each event type, judge the event object described by the descriptive entity phrase, and establish the association relationship between entity phrases; Relationship identification is to determine the relationship between the associated entity phrases, including "event type", "cause" and "status", based on the attribute description dimensions of stop words and phrases.

Images