A water conservancy text-oriented theme evolution relationship mining method

Abstract

The application discloses a kind of subject evolution relationship mining methods for water conservancy text, first, the corpus obtained in advance is preprocessed, and form "text-word matrix" dataset;Second, using LDA model, the data set is clustered, and document-topic matrix and topic-word matrix are generated, to provide support for subsequent theme evolution mining;Then, theme evolution relationship mining is divided into theme interaction relationship mining and theme evolution path mining;Theme interaction relationship mining utilizes the modeling theme evolution model first, focuses on the mutual influence between different theme individuals;Finally, theme evolution path mining utilizes discrete time theme evolution model, and the development context of discipline is bird's eye view from the overall level.The application realizes the accurate and efficient mining of the evolution relationship between different time stages of water conservancy themes, and plays an important role in clarifying the development context of the industry and selecting more promising research areas for researchers.

Description

Technical Field

[0001] This invention belongs to the field of topic mining, specifically relating to a method for mining topic evolution relationships in water conservancy texts. Background Technology

[0002] Topic evolution relationship mining, along with topic mining, belongs to the field of data mining. The former, by mining the topic evolution relationships in text data, can clearly demonstrate the evolution and changes of topics over time, as well as the intersections between different topics, and discover the different stages (such as booms and busts) and evolutionary patterns that topics may involve. Exploring the paths of topic evolution, especially the formation and decline of subject-specific topics concentrated in paper datasets, plays a crucial role in helping researchers understand industry development, discover emerging businesses, and even further predict future evolutionary trends.

[0003] In the field of water conservancy, due to the continuous advancement of water conservancy informatization and smart water conservancy in recent years, an increasing amount of water conservancy literature and data has emerged on the internet. Different users have different focuses on these industry data, and simply relying on industry experts for analysis cannot meet people's needs for information acquisition. Therefore, the industry urgently needs topic evolution relationship mining technology to help people extract information. This is especially true in some interdisciplinary fields, such as smart water conservancy, which integrates multiple disciplines such as water conservancy, environment, computer science, and communications. This characteristic also leads to a large number of topics usually involved in the water conservancy field, and the relationships between different topics are more complex, while also exhibiting strong temporal sequence. Based on the above situation, there is an urgent need for a method for mining water conservancy business topics and evolutionary relationships for text data to help users clarify the industry's development trajectory and select more promising research areas.

[0004] Existing topic evolution mining methods are not comprehensive in uncovering topic evolution relationships. Furthermore, water conservancy business topics involve numerous overlapping areas, resulting in complex and intertwined relationships. Relying on a single topic evolution mining method is often ineffective in these situations. Therefore, to address these issues, this paper proposes a topic relationship classification method that combines the concepts of pre-modeling and discrete time, and improves the topic similarity calculation formula. This method mines water conservancy topics from two aspects: the interrelationships between topics and the topic evolution paths, helping industry professionals clarify the industry's development trajectory and select more promising research areas. Summary of the Invention

[0005] Purpose of the invention: To overcome the shortcomings of the prior art, this invention proposes a method for mining thematic evolutionary relationships in water conservancy texts. It analyzes and mines the relationships from two levels: thematic interaction relationships and thematic evolutionary paths, thereby achieving accurate and efficient mining of the evolutionary relationships between water conservancy themes at different time stages.

[0006] Technical Solution: This invention provides a method for mining topic evolution relationships in water conservancy texts, comprising the following steps:

[0007] (1) Preprocess the pre-acquired corpus to form a “text-word matrix” dataset;

[0008] (2) Use the LDA model to cluster the dataset to generate a document-topic matrix and a topic-word matrix, which will provide support for subsequent topic evolution mining;

[0009] (3) The theme evolution relationship mining is divided into theme interaction relationship mining and theme evolution path mining; theme interaction relationship mining utilizes a theme evolution model to focus on the mutual influence between different theme individuals;

[0010] (4) Thematic evolution path mining: Using discrete time thematic evolution models, we can get a bird's-eye view of the development of the discipline from an overall perspective.

[0011] Further, step (1) includes the following steps:

[0012] (11) Obtain the abstract texts of papers published in Chinese water conservancy journals by web crawling. Each text is stored independently in TXT format, and the file name is named in the form of "time" plus "paper title".

[0013] (12) The acquired corpus is processed by text segmentation, water conservancy field dictionary construction, stop word removal, high-frequency word removal and low-frequency word removal to form a "text-word matrix" dataset; the water conservancy field dictionary construction is achieved through water conservancy official document subject thesaurus, Sogou cell thesaurus and algorithm expansion; the algorithm expansion is that before importing the custom thesaurus, each word item after jieba segmentation constitutes an itemset, each sentence in each document corresponds to a transaction, and the set of all sentences constitutes a database. The association rule mining algorithm is used on it to mine the frequent k-itemsets and association rules formed by the word items; if the rule X→Y is mined, it means that when the premise X of the rule appears, the conclusion Y of the rule also frequently appears; thus, XY is considered to belong to a whole, thereby finding the water conservancy field nouns contained in the document set.

[0014] Furthermore, step (2) includes the following steps:

[0015] (21) Set the number of topics K in the document set, and the parameters of the Dirichlet prior distribution. and

[0016] (22) Randomly assign a topic to each term in each document in the document set;

[0017] (23) Resample a new topic for each term and repeat this process until the sampling converges;

[0018] (24) Analyze the topic assignment of each document in the document set to obtain the topic distribution matrix θ for each document. m ; Analyze the word distribution for each topic to obtain the word distribution matrix d for each topic. k .

[0019] Furthermore, the implementation process of step (3) is as follows:

[0020] (31) Perform topic modeling on the dataset to obtain the document-topic matrix and topic-word matrix of the entire dataset;

[0021] (32) Divide the documents into years according to their time information to obtain the collections D of papers under different time slices. i ={D1,D2,…,D n};

[0022] (33) For the collection of papers in each time slice, calculate the topic popularity relevance and topic co-occurrence of different topics in each time slice; the topic co-occurrence is obtained by calculating the PMI scores of topics that appear simultaneously in the documents to obtain the co-occurrence scores of paired topics:

[0023]

[0024] Among them, P(w i ,w j ) is a word in theme i. i and the word w in theme j j Co-occurrence frequency within a custom window, p(w i ) is a word in theme i. i The higher the PMI, the higher the co-occurrence rate among the themes.

[0025] The calculation process for the topic popularity relevance is as follows:

[0026] The popularity of a topic in a given year is calculated as the ratio of the number of documents related to that topic to the total number of documents in that year.

[0027]

[0028] Among them, P(topic) k |y) represents the popularity of topic k in year y, time d Let P(topic) represent the year document d was published. If the topic distribution of document d includes topic k, then P(topic) = ... k The value of |d) is set to 1, otherwise it is set to 0. y This represents the total number of documents in year y. It serves a normalization function;

[0029] The popularity sequence of any two topics is obtained as P(topic3) = {P(topic1|y1), P(topic1|y2), ..., P(topic1|y3)}. n )} and P(topic2)={P(topic2|y1),P(topic2|y2),…,P(topic2|y n The correlation between two popularity sequences calculated using the Pearson correlation coefficient method is the topic popularity correlation of the topic pair.

[0030]

[0031] in, This represents the average of the popularity series over an equal time period. If Corr(·) = 0, it means that the popularity of the two topics is not related. If the value of Corr(·) is greater than 0, it means that the popularity of the two topics is positively correlated. Similarly, when the value of Corr(·) is less than 0, it means that the popularity of the two topics is negatively correlated.

[0032] (34) Based on the positive or negative correlation of topic popularity and the high or low co-occurrence, the relationship between topics is divided into four categories: promotion relationship, competition relationship, homology relationship, and succession relationship.

[0033] Furthermore, the implementation process of step (4) is as follows:

[0034] (41) Divide the documents into years according to their time information to obtain the collections D of papers under different time slices. i ={D1,D2,…,D n};

[0035] (42) Perform topic modeling on the dataset for each time slice to obtain the document-topic matrix and topic-word matrix for each time slice;

[0036] (43) For topics in adjacent time slices, calculate topic similarity and set a threshold for filtering;

[0037] (44) The evolution path of the topic is presented in the form of a Sankey diagram based on the topic similarity.

[0038] Furthermore, step (43) includes the following steps:

[0039] (431) Obtain the document-topic matrix for each year. and topic-word matrix in, For the document-topic matrix in the i-th time slice, This represents the topic-word matrix for the i-th time slice;

[0040] (432) Using the sliding window technique, the correlation between topics in two adjacent time slices is calculated. In the calculation process, the topic-word matrix is ​​used as the representation of each topic, and the similarity between topics is calculated using the word distribution of topics:

[0041]

[0042] Among them, P ij The similarity between topic i and topic j is represented by an exponent, w. in and w jn Let represent the probability distributions of word n ​​in topic i and topic j, respectively. s represents the number of identical words in the two topic-word distributions, and m represents the total number of words in each topic-word distribution. The more identical words two topic-word distributions contain, the higher the topic similarity. The similarity is also related to the probability distribution of words, and vice versa.

[0043] (433) Set a threshold to filter topic similarity.

[0044] Beneficial Effects: Compared with existing technologies, the present invention offers the following benefits: Most existing topic evolution mining methods analyze evolutionary relationships from a single perspective, lacking a multi-faceted and systematic analysis process for water conservancy topics with overlapping fields. The present invention utilizes the results of topic models to conduct mining and analysis from two levels: topic interaction relationships and topic evolution paths. Specifically, topic interaction relationship mining focuses on the relationships between individual topics, revealing the mutual influence between topics and helping users focus on key topics and promptly grasp decision-making objectives. Topic evolution path mining provides a holistic overview of the discipline's development, helping users clarify industry development trends and uncover emerging businesses. The present invention achieves accurate and efficient mining of evolutionary relationships between water conservancy topics at different time stages, playing a crucial role in helping researchers clarify industry development trends and select more promising research areas. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the framework of the topic evolution relationship mining method described in this invention. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to the accompanying drawings.

[0047] This invention provides a method for mining topic evolution relationships in water conservancy texts, such as... Figure 1 As shown, the specific steps include:

[0048] S1: Preprocess the pre-acquired corpus to form a "text-word matrix" dataset.

[0049] Obtain the abstract texts of papers published in domestic Chinese water conservancy journals through web crawlers. Each text is stored independently in TXT format, and the file name is named in the form of "time" plus "paper title". In the corpus collection and preprocessing part, through the web crawler Scrapy framework, a total of 2,653 abstract texts of papers published in domestic Chinese water conservancy journals such as "Journal of Hydraulic Engineering" and "Water Resources Informatization" are obtained. Then, preprocessing such as word segmentation and noise reduction is performed on the corpus to provide data support for subsequent topic clustering.

[0050] The water conservancy paper collection used in this invention consists of the abstract texts of papers published in domestic Chinese water conservancy journals from 2012 to 2019. The paper sources include journals such as "Journal of Hydraulic Engineering", "Water Resources Informatization", and "People's Yellow River". The paper collection includes 10 categories such as "flood control and prevention", "urban waterlogging", "river and lake treatment", and "water environment".

[0051] Data preprocessing: Perform text word segmentation, construction of a dictionary in the water conservancy field, removal of stop words, removal of high-frequency words, and removal of low-frequency words on the acquired corpus to form a "text-word matrix" dataset. In the word segmentation stage, to avoid incorrect word segmentation caused by traditional word segmentation tools in water conservancy field texts, a domain dictionary library is constructed in this paper to assist word segmentation. The sources of the dictionary library are respectively the water conservancy official document thesaurus; the Sogou cell dictionary; algorithm expansion. The main idea of the algorithm is that before importing the custom dictionary, each word item after jieba word segmentation constitutes an item set. Each sentence in each document corresponds to a transaction, and the set of all sentences constitutes a database. The association rule mining algorithm is used for it, and the frequent k-item sets and association rules composed of word items can be mined. If the rule X→Y is mined, it means that when the premise X of the rule appears, the conclusion Y of the rule also often appears. Therefore, it can be considered that XY belongs to a whole, and thus the proper nouns included in the document set are found. Noise reduction deletes meaningless function words, for example, words such as "de", "di", and "le".

[0052] S2: In the LDA topic model clustering part based on the gensim library, use the LDA model to cluster the dataset to generate a document-topic matrix and a topic-word item matrix, providing support for subsequent topic evolution mining.

[0053] The LDA topic model clustering based on the gensim library employs the implicit Dirichlet approach. It posits that while the topic and term distributions are unknown, they are not uniquely determined but rather random variables following a certain distribution. Therefore, two hyperparameters following Dirichlet prior distributions are introduced: α, the hyperparameter of the prior distribution of the topic distribution (i.e., the Dirichlet distribution), and β, the hyperparameter of the prior distribution of the term distribution (i.e., the Dirichlet distribution). Since the Dirichlet distribution is the conjugate prior distribution of the multinomial distribution, using its conjugate prior distribution simplifies the model's computation for topic and term distributions that follow multinomial distributions. Therefore, LDA chooses the Dirichlet distribution as the prior distribution for these two hyperparameters. The introduction of the Dirichlet prior distribution hyperparameter β makes the LDA model a complete generative model. Specifically, it includes the following steps:

[0054] 1) Set the number of topics in the document set to K=20, the number of words representing the topics to n=10, and the parameters of the Dirichlet prior distribution. and

[0055] 2) Randomly assign a topic to each term in each document in the document set.

[0056] 3) Resample a new topic for each term and repeat this process until the sampling converges.

[0057] 4) Analyze the topic distribution of each document in the document set to obtain the topic distribution matrix θ for each document. m ; Analyze the word distribution for each topic to obtain the word distribution matrix d for each topic. k .

[0058] S3: Mining Inter-Topic Interactions. This involves first modeling a topic evolution model and then focusing on the mutual influence between different topic individuals.

[0059] Employing a modeling-first approach, we first perform topic modeling on the entire dataset, obtaining the document-topic matrix and topic-word matrix for the entire dataset. Then, we divide the documents according to their temporal information, using years as the unit, to obtain the collection D of papers for different time slices. i ={D1,D2,…,D n For each time slice of the paper collection, the popularity relevance and co-occurrence of different topics in each time slice are calculated. Topic co-occurrence is calculated by determining the PMI scores of topics appearing simultaneously in documents to obtain the co-occurrence scores of paired topics. The specific calculation formula is as follows:

[0060]

[0061] Among them, P(w i ,w j ) is a word in theme i. i and the word w in theme j j Co-occurrence frequency within a custom window, p(w i ) is a word in theme i. i As shown in the formula, the higher the PMI, the higher the co-occurrence rate among the themes.

[0062] Topic popularity relevance is defined as a measure of the correlation between the popularity of two topics over different time periods; essentially, it calculates the correlation between the popularity sequences of two topics. Therefore, before calculating topic popularity relevance, topic popularity must first be measured. The popularity of a topic in a given year is calculated as the ratio of the number of documents associated with that topic to the total number of documents in that year. The specific calculation formula is as follows:

[0063]

[0064] Among them, P(topic) k |y) represents the popularity of topic k in year y, time d Let P(topic) represent the year document d was published. If the topic distribution of document d includes topic k, then P(topic) = ... k The value of |d) is set to 1, otherwise it is set to 0. y This represents the total number of documents in year y. It plays a role in normalization.

[0065] Using the formula for calculating topic popularity, we can obtain the popularity sequence P(topic1) = {P(topic1|y1), P(topic1|y2), ..., P(topic1|y3)} for any two topics. n )} and P(topic2)={P(topic2|y1),P(topic2|y2),…,P(topic2|y n The correlation between two popularity sequences is calculated using the Pearson correlation coefficient method, which represents the topic popularity correlation of the topic pair. The specific calculation formula is shown below:

[0066]

[0067] in, This represents the average of the popularity series over an equal time period. The correlation between the popularity of two topics, topic1 and topic2, is determined using Corr(topic1, topic2). If Corr(·) = 0, it means the popularity of the two topics is uncorrelated; when the value of Corr(·) is greater than 0, it indicates a positive correlation between the popularity of the two topics; similarly, when the value of Corr(·) is less than 0, it indicates a negative correlation between the popularity of the two topics.

[0068] Based on the positive or negative correlation and high or low co-occurrence of topics, the relationships between topics are divided into four categories. The definition and classification principles of these relationships are as follows:

[0069] Promotion: If two topics tend to appear together in the same article, have a high degree of co-occurrence, and their popularity is positively correlated over time, then this set of topics is defined as "partner" topics in this paper, and their relationship is a "promotion" relationship. Topics with a "promotion" relationship often appear in the same document and are used together when analyzing a specific idea or area of ​​interest.

[0070] Competition: Topics in a competitive relationship do not tend to appear in the same document simultaneously, have low topic co-occurrence, and their popularity is negatively correlated over time. These topics have nothing in common and are even competitors.

[0071] Homologous: Homologous relationships describe topics that are positively correlated in popularity but do not tend to appear simultaneously in the same document, resulting in low topic co-occurrence. Topics in this relationship tend to evolve in similar patterns over time and may share a common origin, even if they are not directly related.

[0072] Replacement: Replacement is a relationship between topics that tend to appear simultaneously in the same text, exhibiting high co-occurrence but negatively correlated popularity over time. This relationship is crucial for identifying replacement relationships between topics. These topics share a common context within the research field, but as one topic increases, another decreases.

[0073] S4: Propose a topic evolution path mining method that utilizes a discrete-time topic evolution model to provide a bird's-eye view of the development of the discipline from an overall perspective.

[0074] Using the concept of discrete time, the documents are divided into yearly units according to their time information, resulting in a collection D of papers in different time slices. i ={D1,D2,…,D nTopic modeling is performed on the dataset for each time slice, resulting in document-topic matrices and topic-word matrices for each time slice. For topics in adjacent time slices, topic similarity is calculated, and a threshold is set for filtering. The topic evolution path is then represented in the form of a Sankey diagram based on topic similarity.

[0075] Obtain the document-topic matrix for each year. and topic-word matrix in, For the document-topic matrix in the i-th time slice, Let be the topic-word matrix for the i-th time slice. A sliding window technique is used to calculate the topic correlation between topics in adjacent time slices. During the calculation, the topic-word matrix is ​​used as the representation of each topic, and the similarity between topics is calculated using the word distribution of the topics.

[0076]

[0077] Among them, P ij The similarity between topic i and topic j is represented by an exponent, w. in and w jn Let represent the probability distribution of word n ​​in topic i and topic j respectively, s represent the number of identical words in the two topic-word distributions, and m represent the total number of words in each topic-word distribution; the more identical words in two topic-word distributions, the higher the topic similarity, and the similarity is also related to the probability distribution of words, and vice versa; set a threshold to filter topic similarity.

[0078] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. The descriptions herein are merely illustrative of the principles of the invention; various changes and modifications can be made without departing from the spirit and scope of the invention, and all such changes and modifications fall within the scope of the claims. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for mining subject evolution relationship of water-related texts, characterized in that, The method comprises the following steps: (1) preprocessing the pre-acquired corpus to form a "text-word matrix" dataset; (2) using an LDA model to perform clustering processing on the dataset to generate a document-topic matrix and a topic-word matrix, thereby providing support for subsequent topic evolution mining; (3) dividing topic evolution relationship mining into topic interaction relationship mining and topic evolution path mining; the topic interaction relationship mining utilizes a pre-modeling topic evolution model to focus on the mutual influence between different topic individuals; (4) the topic evolution path mining utilizes a discrete-time topic evolution model to bird's-eye view the development context of the discipline from the overall level; The step (1) comprises the following steps: (11) acquiring the paper abstract texts published on domestic Chinese water conservancy journals through a network crawler, each text being independently stored in a TXT format, and the file name being named in the form of "time" plus "paper title"; (12) performing text segmentation, water conservancy field dictionary construction, stop word removal, high-frequency word removal and low-frequency word removal processing on the acquired corpus to form a "text-word matrix" dataset; the water conservancy field dictionary construction is realized through a water conservancy document thesaurus, a sogou cell dictionary and algorithm expansion; wherein the algorithm expansion is that, before importing a self-defined word library, a jieba segmentation is performed on each word item to form a item set, each sentence in each document corresponds to a transaction, and the collection of all sentences constitutes a database; a correlation rule mining algorithm is adopted to mine the frequent J item set and the correlation rule formed by the word items; if a rule X→Y is mined, it is considered that, in the case where the premise X of the rule appears, the conclusion Y of the rule also often appears; thus, it is considered that XY belongs to an integral whole, so as to find the water conservancy field nouns contained in the document set; The step (2) comprises the following steps: (21) setting the number of topics K of the document set and the parameter of the Dirichlet prior distribution and (22) randomly assigning a topic to each word item of each document in the document set; (23) resampling a new topic for each word item, and repeatedly resampling until the sampling converges; (24) Count the topic assignments of each document in the corpus, obtaining the topic distribution matrix θ for each document m ; Count the word distribution of each topic, obtaining the word distribution matrix Q for each topic A ; The step (3) is implemented in the following manner: (31) performing topic modeling on the dataset to obtain a document-topic matrix and a topic-word matrix of the entire dataset; (32) The documents are divided according to their time information in units of years to obtain the paper set D under different time slices h = {D1, D2, …, D n}; (33) calculating the topic popularity correlation and the topic co-occurrence degree of different topics in each time slice for the paper set in each time slice; the topic co-occurrence degree is obtained by calculating the PMI score of the topics appearing in the document at the same time to obtain the co-occurrence score of the pair of topics: where P(w i ,w j ) is the co-occurrence frequency of the word w i in topic i and the word w j in topic j within the custom window, p(w i ) is the occurrence frequency of the word w i in topic i, and PMI is the pointwise mutual information. The greater the PMI, the higher the co-occurrence degree between the topics. The topic popularity correlation is calculated in the following manner: The popularity of the topic in the year is calculated by taking the ratio of the number of documents to which the topic belongs to the total number of documents in the year: where P(topic k |y) represents the popularity of topic k in year y, time D |y) represents the year in which document D was published, and P(topic k |D) is set to 1 if topic k is contained in the topic distribution of document D, and 0 otherwise, C y |y) represents the total number of documents in year y, serves as a normalization factor; P(topic2) = P(topic2\y1), P(topic2\y2),..., P(topic2\y u )} and P(topic2) = P(topic2\y1), P(topic2\y2),..., P(topic2\y u )}; the correlation between the two popularity sequences is the topic-popularity correlation of the topic pair by Pearson correlation coefficient method: wherein, represents the average of the equal-length time popularity sequence; if Corr( ) = 0, it means that the popularity of the two topics is not related; if the value of Corr( ) is greater than 0, it means that the popularity of the two topics is positively correlated, and similarly, when the value of Corr( ) is less than 0, it means that the popularity of the two topics is negatively correlated; (34) according to the positive or negative correlation of the topic popularity and the high or low of the co-occurrence degree, the relationship between the topics is divided into four categories, namely, a promotion relationship, a competitive relationship, a homologous relationship and a alternation relationship; The step (4) is implemented in the following manner: (41) The documents are divided according to their time information in units of years to obtain the paper set D under different time slices i = {D1, D2, …, D n}; (42) performing topic modeling on the dataset in each time slice to obtain a document-topic matrix and a topic-word matrix in each time slice; (43) calculating the topic similarity of the topics in adjacent time slices, and setting a threshold value for filtering; (44) The topic evolution paths are presented in the form of a Sankey diagram according to the topic similarity.

2. The method of claim 1, wherein, The step (43) comprises the following steps: (431)obtaining a document-topic matrix for each year and a topic-word matrix wherein, is a document-topic matrix for the gth time slice, is a topic-word matrix for the gth time slice; (432) The topic similarity between adjacent two time slices is calculated by using the sliding window technique, and the topic-word matrix is used as the representation of each topic in the calculation process, and the topic similarity is calculated by using the word distribution of the topic: where P ij represents the similarity between topic i and topic j is represented by the exponential, w iB and w jB represent the probability distribution of the word B in topic i and topic j, respectively, s represents the number of identical words in the two topic-word distributions, and M represents the total number of words in each topic-word distribution; the more identical words contained in the two topic-word distributions, the higher the similarity of the topics, and the similarity is also related to the probability distribution of the words, and vice versa; (433) The threshold is set to filter the topic similarity.

Images