Disaster information extraction and rapid assessment method based on social media data

Abstract

The application discloses a disaster information extraction and rapid assessment method based on social media data, belonging to the technical field of disaster management and emergency response; the method comprises the following steps: collecting and preprocessing social media data systematically, and performing multidimensional analysis by using advanced natural language processing and geographic information technology. The method covers keyword setting, data crawling and storage, text cleaning, word segmentation and theme clustering, text classification and geographic entity recognition based on deep learning, and finally combines sentiment analysis and physical loss assessment to realize rapid extraction of disaster information and real-time assessment of loss. Through this method, the disaster situation can be grasped in time, scientific decisions can be made by relevant departments, and disaster response capability can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of disaster management and emergency response, and specifically relates to a disaster information extraction and rapid assessment method based on social media data. BACKGROUND

[0002] Currently, disaster information extraction and loss assessment mainly rely on traditional methods such as field investigation, satellite remote sensing image analysis, unmanned aerial vehicle patrol, and historical data statistics. These methods provide data support for disaster management to some extent, but also have significant shortcomings, affecting the efficiency and accuracy of emergency response. On the other hand, with the rapid development of the Internet and social media, social media data is considered as a potential source of disaster information. During a disaster, the public will share real-time information, post for help signals, upload on-site pictures and videos through platforms such as Weibo and WeChat. This provides a new data channel for disaster management. However, existing technologies also face many challenges in using social media data for disaster information extraction.

[0003] 1. Traditional data acquisition methods have insufficient real-time performance

[0004] Field investigation requires professional personnel to enter the disaster area to collect first-hand information. However, after a disaster occurs, damaged transportation and communication facilities may hinder the entry of investigators, delaying data collection. Satellite remote sensing can cover a wide range of areas, but it is greatly affected by weather conditions, such as cloud cover, which can cause image quality to decrease, making it difficult to provide clear ground conditions in a timely manner. Unmanned aerial vehicle patrol is limited by flight radius, battery life and weather conditions, and cannot fully cover the disaster area in a short period of time. Historical data statistics rely on past disaster records and cannot reflect the current actual situation. These factors result in traditional methods being unable to provide timely and accurate data support in the early stages of a disaster, affecting the effectiveness of emergency decision-making.

[0005] 2. High cost of data acquisition and processing

[0006] Traditional methods require a large investment of manpower, material resources and financial resources. Field investigation requires professional teams and safety guarantees, satellite remote sensing requires expensive equipment and data processing technology, and unmanned aerial vehicle patrol requires professional operators and maintenance costs. These high costs limit the application in resource-limited areas or developing countries, making it impossible to achieve comprehensive monitoring and assessment of disasters.

[0007] 3. Social media data quality is uneven and noisy

[0008] The amount of information on social media platforms is vast, but it contains a lot of disaster-unrelated content, rumors, repetitive information, and advertisements. These noisy data increase the difficulty of information extraction and may lead to false disaster assessment. For example, during major events, rumors and false information can spread rapidly, misleading the judgment of emergency response departments.

[0009] 4. Lack of natural language processing techniques for Chinese social media

[0010] Most existing research and applications focus on English platforms such as Twitter and Facebook, and related natural language processing tools and models are also developed based on English corpora. However, Chinese has unique language characteristics such as lack of space segmentation, strong polysemy, and many homophones, making direct application of English processing methods ineffective. In addition, the lack of Chinese disaster corpora also limits the training and optimization of models, making it difficult to accurately capture key information in Chinese text.

[0011] 5. Limited applicability of existing natural language processing models in disaster situations

[0012] Traditional topic models such as LDA (Latent Dirichlet Allocation) require pre-setting the number of topics and are prone to generating difficult-to-explain topics. This limits the flexibility and accuracy of the model when dealing with sudden and dynamic disaster information. Although neural network-based methods improve performance to some extent, they require a large amount of labeled data and are not suitable for the rapid response needs of the early stages of disasters.

[0013] 6. Lack of multi-dimensional and multi-level information fusion analysis

[0014] Existing methods usually only focus on a certain aspect of data analysis, such as topic extraction, sentiment analysis, or geographic information extraction, and fail to combine multiple methods, making it impossible to fully reveal disaster information. This one-sided analysis limits the depth of understanding of disaster situations and fails to fully leverage the potential of social media data in disaster management. SUMMARY

[0015] The purpose of the present invention is to provide a disaster information extraction and rapid assessment method based on social media data, which comprehensively utilizes natural language processing techniques and geographic information techniques to perform multi-dimensional and multi-level analysis on social media data, achieving efficient and accurate extraction of disaster information and rapid assessment of losses.

[0016] To solve the above technical problems, the present invention provides a disaster information extraction and rapid assessment method based on social media data, characterized by the following steps:

[0017] Obtain microblog data;

[0018] Preprocessing the micro-blog data to obtain preprocessed micro-blog data;

[0019] Clustering the preprocessed micro-blog data to obtain a clustering result;

[0020] Screening the clustering result to obtain a disaster loss related cluster;

[0021] Obtaining location information of the disaster loss related cluster to generate a potential disaster point;

[0022] Performing sentiment analysis on the preprocessed micro-blog data corresponding to the potential disaster point to obtain a public emotion space distribution map;

[0023] Performing loss analysis on the preprocessed micro-blog data corresponding to the potential disaster point to obtain a loss distribution map.

[0024] Preferably, the method comprises the following steps:

[0025] Setting a keyword and a time range;

[0026] According to the keyword and the time range, the micro-blog data is obtained in batches through a public API or a web crawler technology of Sina Weibo;

[0027] The micro-blog data is stored in a database.

[0028] Preferably, the preprocessing of the micro-blog data to obtain the preprocessed micro-blog data comprises the following steps:

[0029] The micro-blog data is sequentially subjected to data cleaning, word segmentation, stop word removal and data formatting processing to obtain the preprocessed micro-blog data.

[0030] Preferably, the clustering of the preprocessed micro-blog data to obtain the clustering result comprises the following steps:

[0031] The preprocessed micro-blog data is converted into semantic embedding vectors through a BERT model;

[0032] The semantic embedding vectors are subjected to dimension reduction through a UMAP algorithm to obtain reduced dimension data;

[0033] The reduced dimension data is clustered through an HDBSCAN algorithm;

[0034] The keywords of each cluster are extracted through a c-TF-IDF method to generate a topic description;

[0035] The cosine similarity between topics is calculated, and similar topics are merged.

[0036] Preferably, the keywords of each cluster are extracted through the c-TF-IDF method to generate the topic description, and the method comprises the following steps:

[0037] Based on the c-TF-IDF method, the keywords of each cluster are extracted, and the importance of the words of each cluster is calculated; the calculation formula of the importance is:

[0038]

[0039] In the formula: c-TF-IDF t,c is the importance of the word t in the cluster c; f t,c is the number of occurrences of the word t in the cluster c; ∑ t′ f t′,c represents the total number of occurrences of all words in the cluster c; N is the total number of clusters; n t is the number of clusters containing the word t;

[0040] Preferably, the calculation formula of the cosine similarity between the topics is:

[0041]

[0042] In the formula: k1, k2 represent the vectors of the two topics;

[0043] Two topics with a similarity higher than a preset similarity threshold are judged as similar topics and are merged.

[0044] Preferably, the geographic information of the disaster loss related cluster is obtained, specifically including the following steps:

[0045] Through the Baidu LAC model, the place name information of the preprocessed microblog data in the disaster loss related cluster is obtained;

[0046] The place name information is disambiguated and completed to obtain standardized place names;

[0047] Through the Gaode map API, the standardized place names are converted into latitude and longitude coordinates;

[0048] According to the latitude and longitude coordinates, the geographic information of the potential disaster point is generated.

[0049] Preferably, the microblog data corresponding to the potential disaster point is subjected to sentiment analysis to obtain a public emotion spatial distribution map, specifically including the following steps:

[0050] Through the SKEP model, the microblog data corresponding to the potential disaster point is subjected to sentiment polarity analysis to obtain the sentiment polarity;

[0051] According to the sentiment polarity, the sentiment score and sentiment tendency of the microblog data are calculated;

[0052] According to the geographic information of the potential disaster point and the sentiment score and sentiment tendency of the microblog data, a public emotion spatial distribution map is drawn.

[0053] Preferably, the emotional polarity includes positive emotions and negative emotions;

[0054] The calculation formula of the sentiment score is:

[0055] S=P pos -P neg

[0056] Where: P pos represents the probability of positive emotion, P neg is the probability of negative emotions;

[0057] The emotional tendency is determined based on the emotional score: S ≥ 0 represents positive emotion; S < 0 represents negative emotion.

[0058] Preferably, loss analysis is performed on the pre-processed microblog data corresponding to the potential disaster site to obtain a loss distribution map, which specifically includes the following steps:

[0059] Match the pre-processed microblog data corresponding to the potential disaster-affected points with the disaster dictionary and the severity dictionary to obtain matching results;

[0060] According to the matching results, the loss score of the pre-processed Weibo data is calculated;

[0061] Aggregate the loss scores by region and time to obtain the total loss score of each region in different time periods;

[0062] Through GIS technology, a loss distribution map is generated based on the total loss score of each region in different time periods.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] 1. Strong real-time performance and quick response

[0065] Instant data acquisition: Utilizing the real-time data streams of social media platforms, a large amount of relevant information can be quickly obtained in the early stages of a disaster, filling the gaps in data missing in the early stages of a disaster by traditional methods.

[0066] Rapid processing and analysis: Through automated data collection and processing processes, as well as efficient natural language processing algorithms, rapid extraction and evaluation of disaster information are achieved, significantly shortening the time for information transmission and decision-making.

[0067] 2. High accuracy and improved information quality

[0068] Advanced natural language processing technology: It uses pre-trained models such as BERT and SKEP to conduct in-depth semantic understanding of Chinese text, improving the accuracy of topic extraction, text classification, and sentiment analysis.

[0069] Customized model fine-tuning: The model is fine-tuned and optimized for the specificity of the disaster field, enhancing the ability to identify disaster-related information and reducing the probability of misjudgment and omission.

[0070] Multi-level screening mechanism: Through topic extraction, text classification, and dictionary matching, a large amount of noise data is eliminated, ensuring the reliability of the extracted information.

[0071] 3. Multi-dimensional information fusion, comprehensive evaluation

[0072] Multi-model collaborative application: A variety of natural language processing techniques such as topic extraction, text classification, sentiment analysis, and loss assessment are used to fully exploit the value of social media data.

[0073] Sentiment impact assessment: Through sentiment analysis by the SKEP model, the impact of disasters on public emotions is quantified, providing data support for public opinion management and psychological counseling.

[0074] Physical loss quantification: Based on self-built disaster dictionaries and degree dictionaries, the physical losses caused by disasters are quantitatively evaluated, revealing the damage degree and spatio-temporal distribution of different disaster bodies.

[0075] 4. Suitable for Chinese social media, filling the research gap

[0076] Optimization for Chinese context: The model and method are optimized for Chinese language characteristics and social media text features, solving the difficulties of Chinese natural language processing and improving the application effect on Chinese social media data. In addition, compared to research mainly targeting English social media, the invention targets Chinese platforms such as Sina Weibo, filling the gap in the field of Chinese disaster information extraction.

[0077] 5. Strong scalability, wide applicability

[0078] Method universality: The proposed method and process are universal and can be applied to various types of natural disasters such as floods, earthquakes, and typhoons.

[0079] Model scalability: By adjusting keywords, dictionaries, and model parameters, it can easily adapt to different disaster scenarios and needs, with good scalability.

[0080] Low cost and high efficiency: Using existing social media data and open-source models, without expensive hardware equipment and large amounts of manpower, it reduces costs and improves efficiency.

[0081] 6. Overcoming the shortcomings of existing disaster loss assessment

[0082] Solving the data lag problem: Overcoming the lag in data acquisition and processing of traditional methods, it wins valuable time for emergency response.

[0083] Improved information coverage: Through the extensive user base of social media, more comprehensive disaster information is obtained, especially in areas that traditional methods cannot cover.

[0084] Enhanced information depth and breadth: Not only physical loss information of disasters is obtained, but also public emotions and social impacts are deeply explored, achieving comprehensive assessment of disasters.

[0085] 7. Strong technological innovation

[0086] Introduction of pre-trained large models: BERT, SKEP and other pre-trained large models are applied in disaster information extraction, improving the level of text understanding and information extraction.

[0087] Integration of multiple algorithms: BERTopic topic extraction, BERT text classification, LAC named entity recognition and SKEP sentiment analysis are organically combined to form a complete technical chain.

[0088] Innovative loss assessment method: Based on disaster dictionary and degree dictionary, a new loss quantification method is proposed, which can quickly assess disaster loss and make up for the shortcomings of traditional methods. BRIEF DESCRIPTION OF DRAWINGS

[0089] The specific embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0090] Figure 1 is a flowchart of a disaster information extraction and rapid assessment method based on social media data according to the present application;

[0091] Figure 2 is a potential disaster point and sentiment polarity distribution diagram in Example 1

[0092] Figure 3 is a loss spatial and temporal distribution diagram of various disaster-bearing bodies in Example 1. DETAILED DESCRIPTION

[0093] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in many different ways beyond the specific embodiments described herein, and it is understood that one skilled in the art can make similar modifications without departing from the spirit and scope of the present application, therefore the present application is not limited to the specific implementation disclosed below.

[0094] The terminology used in the description of one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in the description of one or more embodiments of the present specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0095] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these

[0096] The present application will be described in further detail below with reference to the accompanying drawings:

[0097] The present application provides a disaster information extraction and rapid assessment method based on social media data. Through systematic collection and preprocessing of social media data, advanced natural language processing and geographic information technology are used for multidimensional analysis. This method covers keyword setting, data crawling and storage, text cleaning, word segmentation and topic clustering, text classification and geographic entity recognition based on deep learning, and finally combines sentiment analysis and physical loss assessment to realize rapid extraction of disaster information and real-time assessment of loss. Through this method, the disaster situation can be grasped in time, and the relevant departments can make scientific decisions to improve the disaster response capability.

[0098] Compared with the traditional methods relying on field investigation, satellite remote sensing, unmanned aerial vehicle patrol and historical data statistics, this method has the advantages of strong real-time, low cost and wide coverage. Through advanced natural language processing technology, especially for the language characteristics of Chinese social media, noise information is effectively filtered and key disaster data is accurately extracted. In addition, combined with geographic information technology, multidimensional and multilevel information fusion analysis is realized, and disaster dynamics, emotional impact and physical loss situation are quickly revealed. This method overcomes the shortcomings of traditional methods in data acquisition timeliness, high processing cost, uneven information quality and lack of comprehensive analysis capability, and provides more timely decision support for emergency response.

[0099] The present application specifically includes the following steps:

[0100] 1.1 Microblog data collection

[0101] Set keywords and time range: Identify relevant keywords for the disaster event and the time period for data collection.

[0102] Write a crawler program: Use Sina Weibo's public API or web scraping techniques to batch collect relevant microblog data; microblog data includes blog content, posting time, user information, etc.

[0103] Data storage: Store the collected blog content, posting time, user information, etc. into the database.

[0104] 1.2 Data preprocessing, get preprocessed data;

[0105] Data cleaning: Remove duplicate and irrelevant blog posts, filter out noise data.

[0106] Word segmentation: Use Jieba word segmentation tool for Chinese word segmentation.

[0107] Remove stop words: According to the actual data, add frequently appearing words in disaster topics but less contribution to analysis.

[0108] Data formatting: Organize the processed text data into a standard format for subsequent analysis.

[0109] 1.3 Clustering analysis using topic extraction

[0110] Text embedding: Use BERT model to convert text (preprocessed microblog data) into semantic embedding vectors.

[0111] Dimensionality reduction: Use UMAP algorithm to reduce high-dimensional vectors.

[0112] Clustering analysis: Use HDBSCAN algorithm to cluster the reduced data.

[0113] Theme generation: Use c-TF-IDF method to extract keywords for each cluster and generate theme description.

[0114] Theme optimization: Calculate the cosine similarity between themes and merge similar themes.

[0115] 1.4 Text filtering

[0116] Model training: Fine-tune BERT-base-Chinese model based on labeled data to train text classifier.

[0117] Model evaluation: Evaluate model performance through validation set and test set to ensure accuracy and recall rate.

[0118] Text classification: Use the trained model to classify all blog posts and filter out disaster loss-related text.

[0119] 1.5 Named Entity Recognition and Geographical Information Positioning

[0120] Named Entity Recognition: Using Baidu LAC model to extract geographical entity in blog.

[0121] Geographical Name Standardization: Disambiguation and completion of extracted geographical names to obtain standardized geographical names.

[0122] Geocoding: Using Gaode Map API to convert standardized geographical names to latitude and longitude coordinates.

[0123] Generate disaster point list: Organize geographical coordinates to generate geographical information of potential disaster points.

[0124] 1.6 Sentiment Influence Evaluation: Using SKEP model to analyze the sentiment polarity of preprocessed microblog data of potential disaster points.

[0125] Sentiment score calculation: Calculate the sentiment score of each blog.

[0126] Spatial distribution drawing: Draw public emotion spatial distribution map combined with geographical coordinates.

[0127] 1.7 Physical Loss Evaluation

[0128] Dictionary matching: Tokenize blog and match disaster dictionary and degree dictionary.

[0129] Loss score calculation: Calculate the loss score of each blog according to the matching result, and the loss score is the sum of the degree words corresponding to the disaster loss topics in the disaster dictionary.

[0130] Data statistics: Aggregate loss scores by region and time.

[0131] Visualization: Use GIS technology to generate loss distribution map.

[0132] In order to better illustrate the technical effect of the present application, the present application provides the following specific embodiment to illustrate the above technical process:

[0133] Embodiment 1, a disaster information extraction and rapid evaluation method based on social media data, the present application will be further illustrated by actual cases; as shown in the following steps: Figure 1

[0134] 2.1 Data collection

[0135] First, through network crawler technology, collect microblog data related to specific disaster events from social media platform.

[0136] (1) Keyword setting: Determine the keywords related to the target disaster event, such as "heavy rain", "typhoon", "extreme rainfall", etc.​

[0137] (2) Time Range: Set the time range for data collection, from July 28 to August 1, 2023.

[0138] (3) Data Fields: Collect fields such as microblog content, author nickname, publication time, likes, comments, retweets, and location information (such as province, city, IP location).

[0139] (4) Data Storage: Store the collected data in a database for subsequent processing and analysis.

[0140] 2.2 Data Preprocessing

[0141] Preprocess the collected raw data to ensure data quality and analysis accuracy.

[0142] (1) Data Cleaning: Delete duplicate microblogs, remove advertisements and irrelevant content, and filter out possible noise data.

[0143] (2) Text Normalization: Standardize microblog text, including conversion between traditional and simplified Chinese, removal of special characters and emoticons, etc.

[0144] (3) Chinese Word Segmentation: Use word segmentation tools (Jieba word segmentation) to segment microblog text into word sequences.

[0145] (4) Remove Stop Words: Remove commonly used Chinese stop words such as "of", "has", "in", etc., to improve the effectiveness of text analysis.

[0146] 2.3 Topic Extraction and Cluster Analysis

[0147] Use the BERTopic model to extract topics and cluster analysis from preprocessed microblog data, identify major topics of public concern, and provide basis for subsequent loss assessment.

[0148] (1) Text Embedding: Use pre-trained BERT model to convert text into high-dimensional vectors to capture semantic information.

[0149] (2) Dimensionality Reduction: Use UMAP algorithm to reduce the dimensionality of high-dimensional vectors to preserve the global structure of data.

[0150] (3) Density Clustering: Use HDBSCAN algorithm to cluster the reduced data, automatically determine the number of clusters.

[0151] (4) Topic Generation: Based on the c-TF-IDF method, calculate the importance of words for each cluster (class), highlight words that frequently appear in the cluster but rarely appear in the entire corpus, and generate topics, formula as follows:

[0152]

[0153] where f is the number of occurrences of the word t in the cluster c. t,c is the total number of occurrences of all words in the cluster c. N is the total number of clusters. t′ f t′,c is the total number of occurrences of all words in the cluster c. N is the total number of clusters. t is the number of clusters that contain the word t.

[0154] (5) Topic merging and naming: Calculate the similarity between topics, formula as follows:

[0155]

[0156] where k1, k2 represent the vectors of the two topics.

[0157] Set the similarity threshold (0.7), topics above this value are considered similar, combined with manual review, similar topics are merged and named as explicit topic categories, finally form five major topics: "heavy rain description", "emergency rescue", "disaster prayer", "help information", "disaster loss" and so on.

[0158] 2.4 Text screening

[0159] After the initial topic extraction and clustering analysis, although the main topics related to disasters have been identified, there are still a large number of blog posts unrelated to disaster losses. In order to accurately extract text closely related to disaster losses, further screening of blog posts is needed. For this purpose, we built a text classifier based on the BERT-base-Chinese model to perform binary classification on blog posts and screen out text related to disaster losses. This process involves randomly sampling samples from the collected blog posts and manually annotating them, classifying them into two categories: "related to disaster losses" and "unrelated". Then, the annotated dataset is divided into training set, validation set and test set according to the ratio of 6:2:2, ensuring that the training, validation and testing of the model have good representativeness and balance.

[0160] During model training, we use cross-entropy loss function and Adam optimizer to adjust learning rate and batch size and other hyperparameters to optimize the performance of the model. After training, we evaluate the model's accuracy, precision, recall and F1-score on the validation and test sets, as shown in the following table:

[0161] Table 1 Fitting scores on validation and test sets

[0162]

[0163] The results show that the model has high classification performance and can effectively distinguish between disaster loss-related microblog posts. Finally, we used the trained model to classify all microblog posts and successfully screened out 4,451 disaster loss-related texts.

[0164] 2.5 Named Entity Recognition and Geographic Location Mapping

[0165] After screening, the microblog text was subjected to named entity recognition (NER) using the Baidu Lexical Analysis of Chinese (LAC) model to extract potential disaster locations. Next, we used the geographic coding API of Gaode Map to convert standardized place names into specific latitude and longitude coordinates, achieving precise positioning of geographic information. These geographic coordinates were integrated with the corresponding microblog information to generate a list of potential disaster points.

[0166] 2.6 Sentiment Impact Assessment

[0167] To assess the impact of disasters on public sentiment, sentiment polarity analysis was conducted on microblog posts related to potential disaster points. The Baidu-developed sentiment knowledge-enhanced pre-training model SKEP was used to predict the probability of positive and negative emotions for each microblog post under the potential disaster point. Based on the model's output, the sentiment score of each microblog post was calculated:

[0168] Define the sentiment score S:

[0169] S = P pos - P neg

[0170] where P pos represents the probability of positive emotion, and P neg is the probability of negative emotion. Finally, the sentiment tendency is determined according to the sentiment score: S ≥ 0 for positive emotion; S < 0 for negative emotion.

[0171] Combining these sentiment scores with the geographic coordinates of the microblog posts, a spatial distribution map of public sentiment was drawn, as shown in Figure 2 Through analysis of the spatial distribution map, negative emotions are mainly concentrated in areas with severe disasters. This emotional distribution is highly consistent with the actual disaster situation, reflecting the direct impact of disasters on the psychology of local residents. This result provides an important basis for public opinion monitoring for the government and relevant departments, helping to take timely psychological intervention and public opinion guidance measures to maintain social stability.

[0172] 2.7 Disaster Loss Assessment

[0173] Firstly, the disaster-related seed dictionary is constructed according to the keywords in the disaster loss topic in 2.3, including keywords such as "death", "collapse", "submerged" and the like. Then, the seed dictionary is expanded by using the word vector model to form a disaster dictionary containing a large number of related words. At the same time, according to the loss severity described by the words, a degree dictionary is constructed to divide the words into five levels, and the higher the level, the more serious the loss. Finally, the disaster loss dictionary (see Tables 2 and 3, respectively) is formed to quantify the loss of different disaster-bearing bodies and achieve rapid assessment of disaster loss, providing a basis for resource allocation and decision support.

[0174] Table 2 Disaster seed dictionary based on different disaster-bearing bodies

[0175]

[0176] Table 3 Disaster degree dictionary

[0177]

[0178]

[0179] The screened blog posts are matched with the disaster dictionary and the degree dictionary. According to the matching result, the loss score of each blog post is calculated, and the higher the level of the matched words, the higher the loss score. Then, the loss scores are statistically aggregated according to the region and time to obtain the total loss score of each region in different time periods; as shown in Table 4. Figure 3

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, and for example, the division of the modules, modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0181] The units can or can not be physically separated, and the components displayed as units can be one physical unit or multiple physical units, that is, they can be located in one place or distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0182] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.​

[0183] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods of the present disclosure are executed. It should be noted that the above-described computer readable medium of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above.

[0184] The flow charts and block diagrams in the drawings are illustrations of possible architectural, functional, and operational architectures of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow charts or block diagrams can represent a module, a segment, or a portion of code that comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow charts, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0185] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or replacements within the technical scope disclosed by the present disclosure should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for extracting and rapidly assessing disaster information based on social media data, characterized in that: The following steps are involved: Get Weibo data; Preprocessing the Weibo data to obtain preprocessed Weibo data; The pre-processed Weibo data is converted into semantic embedding vectors through the BERT model; The semantic embedding vector is reduced in dimension using the UMAP algorithm to obtain the reduced-dimensional data. The dimensionality-reduced data is clustered using the HDBSCAN algorithm; Through the c-TF-IDF method, the keywords of each cluster are extracted to generate topic descriptions; Calculate the cosine similarity between topics, merge similar topics, and obtain clustering results; The clustering results are screened to obtain clusters related to disaster losses; The Baidu LAC model is used to extract the place name information of the pre-processed Weibo data in the disaster loss related clusters; Disambiguate and complete place name information to obtain standardized place names; Use the Amap API to convert standardized place names into longitude and latitude coordinates; Generate geographic information of potential disaster-affected points based on latitude and longitude coordinates; Through the SKEP model, sentiment polarity analysis is performed on the corresponding Weibo data of potential disaster-affected points to obtain sentiment polarity; According to the sentiment polarity, the sentiment score and sentiment tendency of the Weibo data are calculated; Draw a spatial distribution map of public sentiment based on the geographic information of potential disaster sites and the sentiment scores and sentiment tendencies of Weibo data; Loss analysis is performed on the pre-processed Weibo data corresponding to the potential disaster-stricken points to obtain a loss distribution map.

2. The method for extracting and rapidly assessing disaster information based on social media data according to claim 1, characterized in that: Obtaining Weibo data includes the following steps: Set keywords and time range; Based on keywords and time ranges, batches of Weibo data are obtained through Sina Weibo's public API or web crawler technology; Store microblog data in the database.

3. The method for extracting and rapidly assessing disaster information based on social media data according to claim 2, characterized in that: Preprocessing the microblog data to obtain preprocessed microblog data specifically includes the following steps: The microblog data is processed in sequence by data cleaning, word segmentation, stop word removal and data formatting to obtain the preprocessed microblog data.

4. The method for extracting and rapidly assessing disaster information based on social media data according to claim 3, characterized in that: Using the c-TF-IDF method, keywords for each cluster are extracted to generate topic descriptions. This involves the following steps: Based on the c-TF-IDF method, the keywords of each cluster are extracted and the importance of the words in each cluster is calculated. The calculation formula of the importance is: Where: c-TF-IDF t,c is the importance of word t in cluster c; f t,c is the number of occurrences of word t in cluster c; ∑ t′ f t′,c represents the total number of occurrences of all words in cluster c; N is the total number of clusters; n t is the number of clusters containing word t.

5. The method for extracting and rapidly assessing disaster information based on social media data according to claim 4 is characterized in that: The calculation formula of the cosine similarity between the topics is: Where: k1, k2 represent the vectors of two topics; Two topics with a similarity higher than a preset threshold are considered similar and merged.

6. The method for extracting and rapidly assessing disaster information based on social media data according to claim 5, characterized in that: The emotional polarity includes positive emotions and negative emotions; The calculation formula of the sentiment score is: S=P pos -P neg Where: P pos represents the probability of positive emotion, P neg is the probability of negative emotions; The emotional tendency is determined based on the emotional score: S ≥ 0 represents positive emotion; S < 0 represents negative emotion.

7. The method for extracting and rapidly assessing disaster information based on social media data according to claim 6, characterized in that: Loss analysis is performed on the pre-processed Weibo data corresponding to the potential disaster site to obtain a loss distribution map. The specific steps include the following: Match the pre-processed microblog data corresponding to the potential disaster-affected points with the disaster dictionary and the severity dictionary to obtain matching results; According to the matching results, the loss score of the pre-processed Weibo data is calculated; Aggregate the loss scores by region and time to obtain the total loss score of each region in different time periods; Through GIS technology, a loss distribution map is generated based on the total loss score of each region in different time periods.

Images