A flood situation awareness method and device based on weighted LDA algorithm

Abstract

This invention discloses a flood situation perception method and apparatus based on a weighted LDA algorithm, belonging to the field of flood disaster risk management technology. The method includes: acquiring posts related to flood events through web crawling technology, performing data cleaning and secondary processing to obtain multiple target words and storing them in a dataset; calculating the weight of each target word in the posts using a term frequency-inverse document frequency algorithm; comprehensively determining the number of flood topics using perplexity, consistency index, and Jensen-Shannon divergence; introducing the weights of each target word in the posts into the LDA model, sampling the LDA distribution using a weighted Gibbs sampling algorithm, and estimating the flood post-topic distribution and topic-word distribution to analyze and perceive the flood development trend. This invention can improve the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

Description

Technical Field

[0001] This invention belongs to the field of flood disaster risk management technology, and specifically relates to a flood situation perception method and device based on the weighted LDA algorithm. Background Technology

[0002] With the development of social media platforms (such as Weibo, Facebook, and Twitter), their immediacy has become a crucial channel for obtaining disaster information during emergencies. On the one hand, individuals can use social media to perceive real-time disaster information in virtual cyberspace; on the other hand, each public post is a node, and these nodes constitute an information plane. This real-time social media data can help disaster management personnel better understand the development trend of floods and public opinion, attention, and emotional feelings.

[0003] Current mainstream flood situation awareness methods based on social media data employ topic modeling to identify public attention and spatiotemporal evolution patterns during floods. Topic modeling is a technique for discovering hidden semantic structures within documents, typically resulting in a set of specific topics and keywords. Many clustering algorithms are used for topic modeling, such as k-means and Principal Component Analysis (PCA), but Latent Dirichlet Allocation (LDA) remains the most prevalent method. It allows a word to belong to multiple topics with varying probabilities, rather than simply clustering into clearly defined topics. However, current research using LDA for flood situation awareness modeling based on social media data fails to consider the weight of individual words, leading to decreased accuracy and interpretability in topic identification. This makes it difficult to accurately capture topics related to the flood situation, thus impacting a comprehensive and accurate understanding of the flood situation. Summary of the Invention

[0004] The purpose of this invention is to provide a flood situation awareness method and apparatus based on the weighted LDA algorithm, which aims to improve the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a flood situation awareness method based on a weighted LDA algorithm, comprising the following steps:

[0007] S100: Use web crawler technology to obtain posts related to flood events, perform data cleaning and secondary processing to obtain multiple target words and store them in the dataset;

[0008] S200: Calculate the weight of each target word in the post using the term frequency-inverse document frequency algorithm;

[0009] S300: Determine the number of flood themes by combining perplexity, consistency index and Jensen-Shannon divergence;

[0010] S400: The weights of each target word in the dataset in the posts are introduced into the LDA model. The weighted Gibbs sampling algorithm is used to sample the LDA distribution to estimate the flood post-topic distribution and topic-word distribution in order to analyze the perceived flood development trend.

[0011] Furthermore, in step S100, the data cleaning includes removing erroneous, duplicate, and irrelevant data; the secondary processing includes word segmentation, symbol removal, stop word removal, and hyperlink deletion.

[0012] Further, in step S200, the step of calculating the weight of each target word in the post using the term frequency-inverse document frequency algorithm specifically includes:

[0013] Calculate target words Frequency of appearance in the post tf mn The calculation formula is:

[0014]

[0015] Among them, c mn Indicate target word The number of times it appears in flood posts, with the denominator representing the flood post T. m Sum the occurrences of all words in the text;

[0016] Calculate target words Reverse document frequency (IDF) mn The calculation formula is:

[0017]

[0018] Where |M| represents the total number of flood posts in the corpus, Indicates the presence of the target word The number of flood posts;

[0019] Calculate target words weight The calculation formula is:

[0020]

[0021] Among them, tf mn Indicates word frequency, idfmn Indicates the frequency of reverse document lookup.

[0022] Further, in step S300, the formula for calculating the JSD value in the Jensen-Shannon divergence is:

[0023]

[0024] Where P and Q are two probability distributions, R is a mixed distribution and equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

[0025] Secondly, the present invention provides a flood situation awareness device based on a weighted LDA algorithm, comprising:

[0026] The data acquisition module is used to acquire posts related to flood events through web crawling technology, and to perform data cleaning and secondary processing to obtain multiple target words and store them in the dataset;

[0027] The weight calculation module is used to calculate the weight of each target word in the post using the term frequency-inverse document frequency algorithm.

[0028] The module for determining the number of flood topics is used to comprehensively determine the number of flood topics using perplexity, consistency index, and Jensen-Shannon divergence.

[0029] The modeling and estimation module is used to introduce the weights of each target word in the post into the LDA model. The weighted Gibbs sampling algorithm is used to sample the LDA distribution to estimate the flood post-topic distribution and topic-word distribution in order to analyze the perceived flood development trend.

[0030] Furthermore, the data acquisition module also includes a data cleaning unit and a secondary processing unit; the data cleaning unit is used to remove erroneous, duplicate, and irrelevant data; the secondary processing unit is used for word segmentation, symbol removal, stop word removal, and hyperlink deletion.

[0031] Furthermore, the weight calculation module is specifically used for:

[0032] Calculate target words Frequency of appearance in the post tf mn The calculation formula is:

[0033]

[0034] Among them, c mn Indicate target word The number of times it appears in flood posts, with the denominator representing the flood post T. m Sum the occurrences of all words in the text;

[0035] Calculate target words Reverse document frequency (IDF) mn The calculation formula is:

[0036]

[0037] Where |M| represents the total number of flood posts in the corpus, Indicates the presence of the target word The number of flood posts;

[0038] Calculate target words weight The calculation formula is:

[0039]

[0040] Among them, tf mn Indicates word frequency, idf mn Indicates the frequency of reverse document lookup.

[0041] Furthermore, in the topic quantity determination module, the formula for calculating the JSD value in the Jensen-Shannon divergence is as follows:

[0042]

[0043] Where P and Q are two probability distributions, R is a mixed distribution and equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

[0044] The beneficial effects of this invention are:

[0045] This invention provides a flood situation awareness method and apparatus based on a weighted LDA algorithm. First, it uses web crawling technology to acquire posts related to flood events, and performs data cleaning and secondary processing to obtain multiple target words. Then, it uses a term frequency-inverse document frequency algorithm to calculate the weight of each target word in the posts. Next, it uses perplexity, consistency index, and Jensen-Shannon divergence to comprehensively determine the number of flood topics. Finally, it introduces the weights of each target word in the posts into the LDA model, and uses a weighted Gibbs sampling algorithm to sample the LDA distribution, estimating the flood post-topic distribution and topic-word distribution. This effectively improves the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

[0046] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram illustrating the implementation process of a flood situation awareness method based on the weighted LDA algorithm provided in an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of flood-related perplexity, consistency index, and JSD divergence distance provided in an embodiment of the present invention.

[0050] Figure 3 This is a schematic representation of the theme features for flood situation modeling provided in the embodiments of the present invention.

[0051] Figure 4 This is a functional block diagram of a flood situation awareness device based on the weighted LDA algorithm provided in an embodiment of the present invention.

[0052] Figure reference numerals:

[0053] Data acquisition module 100; Weight calculation module 200;

[0054] The number of topics is determined by a module of 300; the modeling and estimation module has a capacity of 400. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0057] Example 1

[0058] like Figure 1As shown in the figure, a flood situation awareness method based on the weighted LDA algorithm provided by an embodiment of the present invention may include the following steps:

[0059] Step S100: Obtain posts related to the flood event through web crawling technology, perform data cleaning and secondary processing to obtain multiple target words and store them in the dataset.

[0060] In this embodiment, real-time posts related to floods can be obtained from social media platforms (such as Weibo, Facebook, Twitter, etc.) by setting the acquisition period, geographical scope, and keywords. On the other hand, the data cleaning includes removing erroneous, duplicate, and irrelevant data. The secondary processing includes word segmentation, symbol removal, stop word removal, and hyperlink deletion.

[0061] Step S200: Calculate the weight of each target word in the post using the term frequency-inverse document frequency algorithm.

[0062] The term frequency-inverse document frequency (TF-IDF) algorithm is used to calculate the weight of each target word in the flood post. The core idea is that the weight of a word increases proportionally with the number of times it appears in the document, but decreases inversely with the frequency of its appearance in the corpus.

[0063] In one implementation, the specific steps include:

[0064] Calculate target words Frequency of appearance in the post tf mn The calculation formula is:

[0065]

[0066] Among them, c mn Indicate target word The number of times it appears in flood posts, with the denominator representing the flood post T. m Sum the occurrences of all words in the text;

[0067] Calculate target words Reverse document frequency (IDF) mn The calculation formula is:

[0068]

[0069] Where |M| represents the total number of flood posts in the corpus, Indicates the presence of the target word The number of flood posts;

[0070] Calculate target words weight The calculation formula is:

[0071]

[0072] Among them, tf mn Indicates word frequency, idf mn Indicates the frequency of reverse document lookup.

[0073] Step S300: Determine the number of flood themes using a combination of perplexity, consistency index, and Jensen-Shannon divergence.

[0074] Since weighted latent Dirichlet allocation (LDA) is an unsupervised method, the number of topics has a significant impact on the results of flood situation awareness modeling. This embodiment uses perplexity, consistency index, and Jensen-Shannon divergence to comprehensively determine the number of flood topics.

[0075] The Jensen-Shannon divergence measures the semantic similarity between different topic distributions. The formula for calculating the JSD value in the Jensen-Shannon divergence is as follows:

[0076]

[0077] Where P and Q are two probability distributions, R is a mixed distribution and equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

[0078] Step S400: Introduce the weights of each target word in the post into the LDA model, use the weighted Gibbs sampling algorithm to sample the LDA distribution, and estimate the flood post-topic distribution and topic-word distribution to analyze the perceived flood development trend.

[0079] As input, the weighted LDA modeling algorithm requires a text corpus, denoted as . This includes a set of flood posts, T, and a set of feature weights, W. In terms of representation, Here, M corresponds to the number of flood posts, while N is the number of words in the posts, which may vary in different flood posts. This represents the nth word in the mth post. Indicator The corresponding weights. The algorithm's output includes the set θ and For θ m∈θ, which represents the post-topic distribution of the m-th post. Similarly, This represents the topic-word distribution of the k-th topic. In this embodiment, the weighted LDA modeling process is as follows:

[0080] Input: Flood Corpus Number of topics K, m-th flood post - topic weight matrix and cumulative weight matrix δ m The k-th topic-word weight matrix and cumulative weight matrix η k Number of iterations X.

[0081] Output: m-th post - topic distribution θ m and the distribution of the k-th topic-word Estimated value.

[0082] Execution: for 1≤m≤M do

[0083] for each word do

[0084] k←random(0,K);

[0085]

[0086] while x < X do

[0087] for 1≤m≤M do

[0088] for each word do

[0089] ←Weighted Gibbs Sampling

[0090] estimate θ m and

[0091] After weighted Gibbs sampling, θ and θ can be estimated using the following formula.

[0092]

[0093] In the formula, Let represent the probability that the m-th flood post belongs to the k-th topic. This represents the probability that the k-th topic belongs to the υ-th word. Furthermore, the topic distribution of the trajectory can be obtained. For example, θ1 = {Topic 1: 80%, Topic 2: 10%, ...}.

[0094] Similarly, the word distribution of the topic is as follows: For example,

[0095] In one specific implementation, a Python web crawler was used to collect 34,731 flood-related posts from July 1 to July 31. After removing duplicate and invalid posts, 20,962 posts remained and were used for subsequent flood situation awareness modeling based on weighted LDA.

[0096] The original flood posts contained distracting information such as hyperlinks, tags, and symbols, which could affect the results of flood situational awareness modeling. Therefore, this invention uses regular expressions to remove all distracting information and any posts with fewer than three characters. To achieve language simplification and efficient vectorization, this invention uses the Natural Language Toolkit (NLTK) to remove stop words such as "a", "an", "do", and "in". Finally, all cleaned posts are converted to lowercase.

[0097] To determine the optimal number of flood situation modeling topics, such as Figure 2 As shown in (a), the perplexity value decreases with increasing topic number K, reaching an inflection point when K is set to 7. Using the elbow method, it is found that K=7 is close to the optimal number of topics. (b) shows that consistency is high when K is set to 7, 9, or 11. However, (c) and (d) show that when K=9, the JSD divergence distance between topics is small, indicating a degree of similarity between some topics, and the clustering effect is not as good as K=7. Based on these results and the interpretability of each topic, the final number of topics for flood situation modeling is set to 7.

[0098] like Figure 3 As shown, taking a number of topics equal to 7 as an example, principal component analysis (PCA) was used to represent the features of each topic after dimensionality reduction. It can be seen that the feature boundaries of each topic are clearly visible. The results show that the flood situation modeling results are optimal when the number of topics is set to 7.

[0099] The results and keywords related to flood situation modeling were summarized into seven themes. In flood posts, the percentages of themes 1 to 7 were 16.97%, 14.41%, 16.95%, 10.78%, 15.00%, 15.23%, and 10.66%, respectively. The top 15 keywords with the highest relevance to the themes were also listed.

[0100] Theme 1 discusses the macro-level causes of this flood, such as climate change. In the aftermath, scientists and journalists emphasized the link to global extreme weather trends, particularly the more frequent heavy rainfall caused by climate change. Theme 2 showcases public sentiment, with keywords like "prayer," "mourning," "unity," and "heart" expressing deep prayers for the flood victims. Theme 3 focuses on disaster information, with keywords including "missing," "death," "people," "houses," and "cars." Theme 4 emphasizes the impact of the flood. Keywords like "rescue," "search," and "survivors" express public concern about the flood. Theme 5 deals with weather warnings and floods, with the most frequent keywords being "water," "rain," "weather," and "sky," indicating that extreme weather and rainfall triggered flood warnings and forecasts. Theme 6 is somewhat similar to Theme 2; while the public is saddened by the floods, they still hold hope, positive expectations. Theme 7 involves discussions related to flood reporting, with the most frequent keywords being "death," "numbers," "news," "expectations," and "rising." This theme reflects the focus of news reporting on the casualties caused by the floods, a primary concern for the public. The aforementioned thematic modeling results help disaster managers better understand flood development trends and public concerns during flood disasters, which in turn helps in developing disaster mitigation plans and improving urban disaster preparedness and resilience.

[0101] Example 2

[0102] like Figure 4 As shown in the figure, a flood situation awareness device based on the weighted LDA algorithm provided in this embodiment of the invention may include a data acquisition module 100, a weight calculation module 200, a topic quantity determination module 300, and a modeling and estimation module 400.

[0103] In this embodiment, the data acquisition module 100 is used to acquire posts related to the flood event through web crawler technology, and perform data cleaning and secondary processing to obtain multiple target words and store them in the dataset.

[0104] The weight calculation module 200 is used to calculate the weight of each target word in the post using the word frequency-inverse document frequency algorithm.

[0105] The topic quantity determination module 300 is used to determine the number of flood topics by comprehensively utilizing perplexity, consistency index and Jensen-Shannon divergence.

[0106] The modeling and estimation module 400 is used to introduce the weights of each target word in the post into the LDA model, and to sample the LDA distribution using the weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution in order to analyze and perceive the development trend of the flood.

[0107] The device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0108] In summary, the flood situation awareness method and apparatus based on the weighted LDA algorithm provided by this invention first acquires posts related to flood events through web crawling technology, and performs data cleaning and secondary processing to obtain multiple target words. Then, the weight of each target word in the posts is calculated using the term frequency-inverse document frequency algorithm. Next, the number of flood topics is determined by comprehensively using perplexity, consistency index, and Jensen-Shannon divergence. Finally, the weight of each target word in the posts is introduced into the LDA model, and the LDA distribution is sampled using the weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution. This can effectively improve the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

[0109] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A flood situation awareness method based on the weighted LDA algorithm, characterized in that, Includes the following steps: S100: Use web crawler technology to obtain posts related to flood events, perform data cleaning and secondary processing to obtain multiple target words and store them in the dataset; S200: Calculate the weight of each target word in the post using the term frequency-inverse document frequency algorithm; In step S200, the step of calculating the weight of each target word in the post using the term frequency-inverse document frequency algorithm specifically includes: Calculate target words Frequency of appearance in the post tf mn The calculation formula is: Among them, c mn Indicate target word The number of times the flood appears in the m-th flood post, where the denominator represents the number of flood posts T. m Sum the occurrences of all words in the text; Calculate target words Reverse document frequency (IDF) mn The calculation formula is: Where |M| represents the total number of flood posts in the corpus, Indicates the presence of the target word The number of flood posts; Calculate target words weight The calculation formula is: Among them, tf mn Indicates word frequency, idf mn Indicates the frequency of reverse document lookups; S300: Determine the number of flood themes by combining perplexity, consistency index and Jensen-Shannon divergence; In step S300, the formula for calculating the JSD value in the Jensen-Shannon divergence is as follows: Where P and Q are two probability distributions, R is a mixture distribution and equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions; S400: The weights of each target word in the dataset in the posts are introduced into the LDA model. The weighted Gibbs sampling algorithm is used to sample the LDA distribution to estimate the flood post-topic distribution and topic-word distribution in order to analyze the perceived flood development trend. Step S400 includes: Input parameter definition: Input flood corpus Number of topics K, Post-topic weight matrix Cumulative weight matrix δ m The k-th topic-word weight matrix and cumulative weight matrix η k And the number of iterations X, where M is the total number of flood posts and N is the number of words in the posts; Weighting of target words; Weighted Gibbs sampling iteration: for each target word in each flood post When sampling topic allocation, the post-topic weight matrix is ​​updated by combining the target keyword weights. and the k-th topic-word weight matrix Until the iteration count X is reached, the update method is as follows: Through the updated weight matrix and Estimate the topic distribution θ and keyword distribution of posts 2. The flood situation awareness method based on the weighted LDA algorithm according to claim 1, characterized in that, In step S100, the data cleaning includes removing erroneous, duplicate, and irrelevant data; the secondary processing includes word segmentation, symbol removal, stop word removal, and hyperlink deletion.

3. A flood situation awareness device based on the weighted LDA algorithm, characterized in that, The device includes: The data acquisition module is used to acquire posts related to flood events through web crawling technology, and to perform data cleaning and secondary processing to obtain multiple target words and store them in the dataset; The weight calculation module is used to calculate the weight of each target word in the post using the term frequency-inverse document frequency algorithm. The weight calculation module is specifically used for: Calculate target words Frequency of appearance in the post tf mn The calculation formula is: Among them, c mn Indicate target word The number of times it appears in flood posts, with the denominator representing the flood post T. m Sum the occurrences of all words in the text; Calculate target words Reverse document frequency (IDF) mn The calculation formula is: Where |M| represents the total number of flood posts in the corpus, Indicates the presence of the target word The number of flood posts; Calculate target words weight The calculation formula is: Among them, tf mn Indicates word frequency, idf mn Indicates the frequency of reverse document lookups; The module for determining the number of flood topics is used to comprehensively determine the number of flood topics using perplexity, consistency index, and Jensen-Shannon divergence. In the topic quantity determination module, the formula for calculating the JSD value in the Jensen-Shannon divergence is as follows: Where P and Q are two probability distributions, R is a mixture distribution and equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions; The modeling and estimation module is used to introduce the weights of each target word in the post into the LDA model, and to sample the LDA distribution using the weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution in order to analyze the perceived flood development trend. Input parameter definition: Input flood corpus Number of topics K, Post-topic weight matrix Cumulative weight matrix δ m The k-th topic-word weight matrix and cumulative weight matrix η k And the number of iterations X, where M is the total number of flood posts and N is the number of words in the posts; Weighting of target words; Weighted Gibbs sampling iteration: for each target word in each flood post When sampling topic allocation, the post-topic weight matrix is ​​updated by combining the target keyword weights. and the k-th topic-word weight matrix Until the iteration count X is reached, the update method is as follows: Through the updated weight matrix and Estimate the topic distribution θ and keyword distribution of posts 4. The flood situation awareness device based on the weighted LDA algorithm according to claim 3, characterized in that, The data acquisition module further includes a data cleaning unit and a secondary processing unit; the data cleaning unit is used to remove erroneous, duplicate, and irrelevant data; the secondary processing unit is used for word segmentation, symbol removal, stop word removal, and hyperlink deletion.

Images