Flood situation awareness method and device based on weighted LDA algorithm

Abstract

The invention discloses a flood situation sensing method and device based on a weighted LDA algorithm, and belongs to the technical field of flood disaster risk management. The method comprises the following steps: acquiring a post related to a flood event through a web crawler technology, performing data cleaning and secondary processing to obtain a plurality of target words, and storing the target words in a data set; calculating the weight of each target word in the post by using a word frequency-reverse document frequency algorithm; comprehensively determining the quantity of flood themes by utilizing the confusion degree, the consistency index and the Jensen-Shannon divergence; and introducing the weight of each target word in the data set in the post into an LDA model, sampling LDA distribution by using a weighted Gibbs sampling algorithm, and estimating flood post-topic distribution and topic-word distribution so as to analyze and perceive the flood development situation. According to the method, the accuracy and interpretability of theme identification can be improved, so that comprehensive perception and accurate understanding of the flood situation are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of flood disaster risk management, and in particular, relates to a flood situation perception method and device based on a weighted LDA algorithm. Background Art

[0002] With the development of social media platforms (such as Weibo, Facebook, Twitter, etc.), their immediacy has become an important channel for obtaining disaster information in emergencies. On the one hand, individuals can use social media to perceive real-time disaster information in the virtual network space; on the other hand, each public post is a node, and each node constitutes an information plane. These real-time social media data can help disaster management personnel better perceive the development of floods and the public's opinions, concerns, and emotional feelings.

[0003] The current mainstream flood situation awareness method based on social media data is to obtain the public's attention points and temporal and spatial evolution laws during floods through topic modeling. Topic modeling is a technology that discovers hidden semantic structures in documents, and its results are usually a series of special topics and keywords. There are many clustering algorithms currently used for topic modeling, such as k-means, principal component analysis (PCA), etc., but latent Dirichlet allocation (LDA) is still the most mainstream topic modeling method, which allows a word to belong to multiple topics at the same time with different probabilities, rather than only clustering out clearly demarcated topics. However, in the current research on flood situation awareness modeling based on social media data using LDA, the weight of a single word is not considered, resulting in a decrease in the accuracy of topic identification and poor interpretability, making it difficult to accurately capture topics related to the flood situation, thereby affecting the comprehensive perception and accurate understanding of the flood situation. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a flood situation perception method and device based on the weighted LDA algorithm, aiming to improve the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a flood situation awareness method based on a weighted LDA algorithm, comprising the following steps:

[0007] S100: Obtain posts related to flood events through web crawler technology, perform data cleaning and secondary processing, obtain multiple target words and store them in the data set;

[0008] S200: Calculate the weight of each target word in the post using the word frequency-inverse document frequency algorithm;

[0009] S300: Comprehensively determine the number of flood topics using perplexity, consistency index and Jensen-Shannon divergence;

[0010] S400: The weights of each target word in the data set in the post are introduced into the LDA model, and the LDA distribution is sampled using the weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution to analyze and perceive the development trend of the flood.

[0011] Furthermore, in the step S100, the data cleaning includes removing errors, duplications and data irrelevant to the subject; the secondary processing includes word segmentation, symbol removal, stop word removal and hyperlink deletion operations.

[0012] Furthermore, in the step S200, the step of calculating the weight of each target word in the post using the word frequency-inverse document frequency algorithm specifically includes:

[0013] Calculate target word Frequency of occurrence in the post tf mn , the calculation formula is:

[0014]

[0015] Among them, c mn Indicates the target word The number of times it appears in flood posts. The denominator represents the number of flood posts T m Sum the number of occurrences of all words in ;

[0016] Calculate target word The inverse document frequency (idf) mn , the calculation formula is:

[0017]

[0018] Among them, |M| represents the total number of flood posts in the corpus, Indicates that it contains the target word Number of flood posts;

[0019] Calculate target word Weight The calculation formula is:

[0020]

[0021] Among them, tf mn Indicates word frequency, idfmn Represents the inverse document frequency.

[0022] Furthermore, in step S300, the calculation formula of the JSD value in the Jensen-Shannon divergence is:

[0023]

[0024] Among them, P and Q are two probability distributions, M is a mixed distribution and is equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

[0025] In a second aspect, the present invention provides a flood situation awareness device based on a weighted LDA algorithm, comprising:

[0026] The data acquisition module is used to obtain posts related to flood events through web crawler technology, and perform data cleaning and secondary processing to obtain multiple target words and store them in the data set;

[0027] A weight calculation module is used to calculate the weight of each target word in the post using a word frequency-inverse document frequency algorithm;

[0028] The topic number determination module is used to comprehensively determine the number of flood topics using perplexity, consistency index and Jensen-Shannon divergence;

[0029] The modeling and estimation module is used to introduce the weights of each target word in the post in the dataset into the LDA model, sample the LDA distribution using the weighted Gibbs sampling algorithm, and estimate the flood post-topic distribution and topic-word distribution to analyze and perceive the development trend of the flood.

[0030] Furthermore, the data acquisition module also includes a data cleaning unit and a secondary processing unit; the data cleaning unit is used to remove errors, duplications and data irrelevant to the subject; the secondary processing unit is used to segment words, remove symbols, remove stop words and delete hyperlinks.

[0031] Furthermore, the weight calculation module is specifically used to:

[0032] Calculate target word Frequency of occurrence in the post tf mn , the calculation formula is:

[0033]

[0034] Among them, c mn Indicates the target word The number of times it appears in flood posts. The denominator represents the number of flood posts T m Sum the number of occurrences of all words in ;

[0035] Calculate target word The inverse document frequency (idf) mn , the calculation formula is:

[0036]

[0037] Among them, |M| represents the total number of flood posts in the corpus, Indicates that it contains the target word Number of flood posts;

[0038] Calculate target word Weight The calculation formula is:

[0039]

[0040] Among them, tf mn Indicates word frequency, idf mn Represents the inverse document frequency.

[0041] Furthermore, in the topic quantity determination module, the calculation formula of the JSD value in the Jensen-Shannon divergence is:

[0042]

[0043] Among them, P and Q are two probability distributions, M is a mixed distribution and is equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

[0044] The beneficial effects of the present invention are:

[0045] The present invention provides a flood situation perception method and device based on weighted LDA algorithm. Firstly, posts related to flood events are obtained by web crawler technology, and data cleaning and secondary processing are performed to obtain multiple target words. Then, the weight of each target word in the post is calculated by using word frequency-inverse document frequency algorithm. Then, the number of flood topics is comprehensively determined by using perplexity, consistency index and Jensen-Shannon divergence. Finally, the weight of each target word in the post is introduced into the LDA model, and the LDA distribution is sampled by using weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution, which can effectively improve the accuracy and interpretability of topic identification, thereby realizing comprehensive perception and accurate understanding of the flood situation.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is a schematic diagram of the implementation flow of a flood situation awareness method based on a weighted LDA algorithm provided in an embodiment of the present invention.

[0049] Figure 2 It is a schematic diagram of flood topic perplexity, consistency index and JSD divergence distance provided by an embodiment of the present invention.

[0050] Figure 3 This is a representation diagram of the flood situation modeling theme features provided by an embodiment of the present invention.

[0051] Figure 4 It is a functional block diagram of a flood situation awareness device based on a weighted LDA algorithm provided by an embodiment of the present invention.

[0052] Reference numerals in the figures:

[0053] Data acquisition module 100; Weight calculation module 200;

[0054] Topic quantity determination module 300; Modeling estimation module 400. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention.

[0056] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0057] Example 1

[0058] like Figure 1As shown, a flood situation awareness method based on a weighted LDA algorithm provided by an embodiment of the present invention may include the following steps:

[0059] Step S100: Obtain posts related to flood events through web crawler technology, perform data cleaning and secondary processing, obtain multiple target words and store them in a data set.

[0060] In this embodiment, web crawler technology can be used to set the acquisition cycle, acquire geographical range, and acquire keywords from social media platforms (such as Weibo, Facebook, Twitter, etc.) to acquire real-time posts related to floods. On the other hand, the data cleaning includes removing errors, duplications, and data irrelevant to the subject. The secondary processing includes word segmentation, symbol removal, stop word removal, and hyperlink deletion operations.

[0061] Step S200: Calculate the weight of each target word in the post using the word frequency-inverse document frequency algorithm.

[0062] Among them, the term frequency-inverse document frequency (TF-IDF) algorithm is used to calculate the weight of each target word in the flood posts. The core idea is that the weight of a word increases in proportion to the number of times it appears in the document, but at the same time decreases inversely with its frequency of appearance in the corpus.

[0063] In one embodiment, the method specifically comprises the following steps:

[0064] Calculate target word Frequency of occurrence in the post tf mn , the calculation formula is:

[0065]

[0066] Among them, c mn Indicates the target word The number of times it appears in flood posts. The denominator represents the number of flood posts T m Sum the number of occurrences of all words in ;

[0067] Calculate target word The inverse document frequency (idf) mn , the calculation formula is:

[0068]

[0069] Among them, |M| represents the total number of flood posts in the corpus, Indicates that it contains the target word Number of flood posts;

[0070] Calculate target word Weight The calculation formula is:

[0071]

[0072] Among them, tf mn Indicates word frequency, idf mn Represents the inverse document frequency.

[0073] Step S300: Determine the number of flood topics comprehensively using perplexity, consistency index and Jensen-Shannon divergence.

[0074] Since weighted LDA (Weighted Latent Dirichlet allocation) is an unsupervised method, the number of topics has a significant impact on the flood situation awareness modeling results. This embodiment uses perplexity, consistency index and Jensen-Shannon divergence to comprehensively determine the number of flood topics.

[0075] Among them, Jensen-Shannon divergence is used to measure the semantic similarity between different topic distributions. The calculation formula of the JSD value in the Jensen-Shannon divergence is:

[0076]

[0077] Among them, P and Q are two probability distributions, M is a mixed distribution and is equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

[0078] Step S400: Introduce the weights of each target word in the post in the data set into the LDA model, use the weighted Gibbs sampling algorithm to sample the LDA distribution, estimate the flood post-topic distribution and topic-word distribution, and analyze the perceived flood development trend.

[0079] As input, the weighted LDA modeling algorithm requires a text corpus, denoted as It contains the set of flood posts T and the set of feature weights W. In terms of representation, Here, M corresponds to the number of flood posts, and N is the number of words in a post, which may vary in different flood posts. represents the nth word in the mth post, Expressive words The output of the algorithm includes the set θ and For θ m∈θ, which represents the post-topic distribution of the mth post. Similarly, represents the topic-word distribution of the kth topic. In this embodiment, the process of weighted LDA modeling is as follows:

[0080] Input: Flood Corpus The number of topics K, the mth flood post-topic weight matrix and the cumulative weight matrix δ m , the kth topic-word weight matrix and the cumulative weight matrix η k , number of iterations X.

[0081] Output: mth post-topic distribution θ m and the kth topic-word distribution Estimated value.

[0082]

[0083]

[0084] After weighted Gibbs sampling, θ and θ can be estimated using the following formula:

[0085]

[0086] In the formula, represents the probability that the mth flood post belongs to the kth topic, represents the probability of the vth word in the kth topic. In addition, the topic distribution of the trajectory θ = For example, θ1 = {Subject 1: 80%, Subject 2: 10%, ...}.

[0087] Similarly, the word distribution of the topic is For example,

[0088] In a specific implementation, taking the flood disaster in Europe in July 2021 as an example, using "German flood, Belgian flood, Dutch flood" as keywords, a Python crawler was used to collect 34,731 flood posts related to European floods from July 1 to July 31. After removing duplicate and invalid posts, the remaining 20,962 posts were used for subsequent flood situation awareness modeling based on weighted LDA.

[0089] The original flood posts contain interference information such as hyperlinks, tags, symbols, etc., which will affect the flood situation awareness modeling results. Therefore, this patent uses regular expressions to delete all interference information and any posts with less than three characters. In order to achieve language simplification and efficient vectorization, this patent uses the Natural Language Toolkit (NLTK) to remove stop words such as "a", "an", "do", and "in". Finally, all cleaned posts are converted to lowercase letters.

[0090] In order to determine the optimal number of flood situation modeling topics, such as Figure 2 As shown, in Figure 2 In a, the perplexity value decreases as the number of topics K increases, and an inflection point appears when K is set to 7. Using the elbow method, it is found that K = 7 is close to the optimal number of topics. Figure 2 b shows that the consistency is high when K is set to 7, 9, or 11. However, Figure 2 c and 2d show that when K = 9, the JSD divergence distance between topics is small, some topics have a certain degree of similarity, and the clustering effect is not as good as K = 7. Based on the above results and the interpretability of each topic, the number of flood situation modeling topics was finally set to 7.

[0091] like Figure 4 As shown in the figure, taking the number of topics equal to 7 as an example, principal component analysis (PCA) is used to represent the features of each topic after dimensionality reduction. It can be seen that the feature boundaries of each topic are clearly visible. The results show that when the number of topics is set to 7, the result of flood situation modeling is optimal.

[0092] Among them, the topic results and keywords about European flood situation modeling are summarized into 7 topics in total. In the flood posts, topics 1 to 7 account for 16.97%, 14.41%, 16.95%, 10.78%, 15.00%, 15.23% and 10.66% respectively, and the top 15 keywords with the highest relevance to the topic are listed.

[0093] Theme 1 discussed the macro causes of the flood, such as climate change. After the flood, scientists and journalists stressed the connection with global extreme weather trends, especially more frequent heavy rainfall caused by climate change. Theme 2 showed the public's emotions, with the keywords "prayer", "mourning", "solidarity" and "heart" expressing people's deep prayers for the victims of the flood. Theme 3 focused on the disaster information, with some keywords "missing", "death", "people", "houses" and "cars". Theme 4 emphasized the impact of the flood on Germany, which was the country most affected by the flood. The keywords "rescue", "search" and "survivors" expressed people's concern about the flood. Theme 5 was weather warnings and floods. The most frequently appearing keywords were "water", "rain", "weather", "sky" and "China". This theme showed that extreme weather and rainfall triggered flood warnings and forecasts. Theme 6 was similar to Theme 2 to some extent. Although the public was sad about the flood, they still had hope, good and positive expectations. Theme 7 involved discussions related to the coverage of the European floods, with the most frequently appearing keywords being "death", "number of people", "news", "forecast", "Europe" and "rising". This theme reflects the focus of news reports on casualties caused by floods, which is also a major issue of concern to the public. These topic modeling results can help disaster managers better understand the development of floods and the focus of public attention during flood disasters, which will help formulate disaster reduction plans and improve the city's disaster prevention and mitigation resilience.

[0094] Example 2

[0095] like Figure 4 As shown, a flood situation awareness device based on a weighted LDA algorithm provided by an embodiment of the present invention may include a data acquisition module 100 , a weight calculation module 200 , a topic quantity determination module 300 and a modeling estimation module 400 .

[0096] In this embodiment, the data acquisition module 100 is used to acquire posts related to flood events through web crawler technology, and perform data cleaning and secondary processing to obtain multiple target words and store them in a data set.

[0097] The weight calculation module 200 is used to calculate the weight of each target word in the post using a word frequency-inverse document frequency algorithm.

[0098] The topic quantity determination module 300 is used to comprehensively determine the number of flood topics by using perplexity, consistency index and Jensen-Shannon divergence.

[0099] The modeling estimation module 400 is used to introduce the weights of each target word in the post in the data set into the LDA model, use the weighted Gibbs sampling algorithm to sample the LDA distribution, and estimate the flood post-topic distribution and topic-word distribution to analyze the perceived flood development trend.

[0100] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0101] In summary, the flood situation perception method and device based on the weighted LDA algorithm provided by the embodiment of the present invention first obtain posts related to flood events through web crawler technology, and perform data cleaning and secondary processing to obtain multiple target words; then, the weight of each target word in the post is calculated using the word frequency-inverse document frequency algorithm; then, the perplexity, consistency index and Jensen-Shannon divergence are used to comprehensively determine the number of flood topics; finally, the weight of each target word in the post is introduced into the LDA model, and the LDA distribution is sampled using the weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution, which can effectively improve the accuracy and interpretability of topic identification, thereby achieving comprehensive perception and accurate understanding of the flood situation.

[0102] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A flood situation awareness method based on weighted LDA algorithm, characterized in that: The following steps are involved: S100: Obtain posts related to flood events through web crawler technology, perform data cleaning and secondary processing, obtain multiple target words and store them in the data set; S200: Calculate the weight of each target word in the post using the word frequency-inverse document frequency algorithm; S300: Comprehensively determine the number of flood topics using perplexity, consistency index and Jensen-Shannon divergence; S400: The weights of each target word in the data set in the post are introduced into the LDA model, and the LDA distribution is sampled using the weighted Gibbs sampling algorithm to estimate the flood post-topic distribution and topic-word distribution to analyze and perceive the development trend of the flood.

2. The flood situation awareness method based on the weighted LDA algorithm according to claim 1 is characterized in that: In the step S100, the data cleaning includes removing errors, duplications and data irrelevant to the subject; the secondary processing includes word segmentation, symbol removal, stop word removal and hyperlink deletion operations.

3. The flood situation awareness method based on weighted LDA algorithm according to claim 1 is characterized in that: In step S200, the step of calculating the weight of each target word in the post using the word frequency-inverse document frequency algorithm specifically includes: Calculate target word Frequency of occurrence in posts tf mn , the calculation formula is: Among them, c mn Indicates the target word The number of times it appears in flood posts. The denominator represents the number of flood posts T m Sum the number of occurrences of all words in ; Calculate target word The inverse document frequency (idf) mn , the calculation formula is: Among them, |M| represents the total number of flood posts in the corpus, Indicates that it contains the target word Number of flood posts; Calculate target word Weight The calculation formula is: Among them, tf mn Indicates word frequency, idf mn Represents the inverse document frequency.

4. The flood situation awareness method based on the weighted LDA algorithm according to claim 1 is characterized in that: In step S300, the calculation formula of the JSD value in the Jensen-Shannon divergence is: Among them, P and Q are two probability distributions, M is a mixed distribution and is equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

5. A flood situation awareness device based on weighted LDA algorithm, characterized in that: The device comprises: The data acquisition module is used to obtain posts related to flood events through web crawler technology, and perform data cleaning and secondary processing to obtain multiple target words and store them in the data set; A weight calculation module is used to calculate the weight of each target word in the post using a word frequency-inverse document frequency algorithm; The topic number determination module is used to comprehensively determine the number of flood topics using perplexity, consistency index and Jensen-Shannon divergence; The modeling and estimation module is used to introduce the weights of each target word in the post in the dataset into the LDA model, sample the LDA distribution using the weighted Gibbs sampling algorithm, and estimate the flood post-topic distribution and topic-word distribution to analyze and perceive the development trend of the flood.

6. The flood situation awareness device based on the weighted LDA algorithm according to claim 5 is characterized in that: The data acquisition module also includes a data cleaning unit and a secondary processing unit; the data cleaning unit is used to remove errors, duplications and data irrelevant to the subject; the secondary processing unit is used to segment words, remove symbols, remove stop words and delete hyperlinks.

7. The flood situation awareness device based on the weighted LDA algorithm according to claim 5 is characterized in that: The weight calculation module is specifically used for: Calculate target word Frequency of occurrence in posts tf mn , the calculation formula is: Among them, c mn Indicates the target word The number of times it appears in flood posts. The denominator represents the number of flood posts T m Sum the number of occurrences of all words in ; Calculate target word The inverse document frequency (idf) mn , the calculation formula is: Among them, |M| represents the total number of flood posts in the corpus, Indicates that it contains the target word Number of flood posts; Calculate target word Weight The calculation formula is: Among them, tf mn Indicates word frequency, idf mn Represents the inverse document frequency.

8. The flood situation awareness device based on weighted LDA algorithm according to claim 5, characterized in that: In the topic quantity determination module, the calculation formula of the JSD value in the Jensen-Shannon divergence is: Among them, P and Q are two probability distributions, M is a mixed distribution and is equal to 1 / 2(P+Q); D measures the KL divergence between the two probability distributions.

Images