A Method and System for Retrieving Text in the Water Resources Field Based on Topic Mining

Abstract

This invention discloses a method and system for retrieving text in the water resources field based on topic mining. The method includes the following steps: collecting a dataset; preprocessing the dataset; constructing a GAN-BiGRU Topic Attention Model, which combines a bidirectional adversarial neural network (BAN) and a bidirectional GRU (GRU) based on topic mining; validating and testing on a test set; ranking topics by relevance; and performing topic retrieval. The system includes a data crawling module, an index building module, and a data retrieval module. This invention significantly outperforms existing methods in terms of accuracy and recall for topic diversity detection, topic coherence detection, and downstream classification tasks, providing a new solution for research in this field. Based on ample domain data reserves, this invention pioneers the application of a network model combining a BAN and a bidirectional GRU in water resources retrieval, contributing to the application of topic model retrieval in the water resources field.

Description

Technical Field

[0001] This invention relates to natural language processing and information retrieval, and in particular to a method and system for retrieving text in the field of water conservancy based on topic mining. Background Technology

[0002] We live in an information age. With the widespread use of computers, big data, cloud computing, and artificial intelligence in the information field, the digitization of various documents and materials has had a profound impact on traditional information retrieval methods. Therefore, information retrieval technology and related technologies remain widely used and continue to evolve and innovate. In modern times, my country's water conservancy industry has flourished, accumulating a vast amount of textual data in related fields. Consequently, the problem of information overload in water conservancy has become increasingly severe, making it increasingly difficult for users to retrieve the information they need from massive amounts of data.

[0003] In existing research, Bayesian probabilistic topic models (BPTM) are the most popular and successful traditional model family, with the Latent Dirichlet Distribution (LDA) being the most well-known representative. BPTM typically specifies a probabilistic generative model that generates document data with a latent variable structure sampled from a pre-specified distribution connected by Bayes' theorem, capturing topics by these latent variables. Like other Bayesian models, BPTM learning is accomplished through a Bayesian inference process. However, this method still has the following limitations: First, the inference process needs to be customized; the inference complexity increases significantly with model complexity; and the design of the inference process is difficult to automate. Second, this method is difficult to effectively scale on large text datasets and utilize GPUs for parallel computation. Third, this method lacks scalability and cannot be jointly trained with other deep neural networks. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a method and system for retrieving water conservancy-related texts based on topic mining, thereby improving the accuracy and recall of the model in terms of topic diversity, topic coherence, and downstream classification tasks, and ultimately achieving high-efficiency and high-accuracy water conservancy-related text retrieval.

[0005] Technical Solution: The present invention provides a method for retrieving text in the water conservancy field based on topic mining, comprising the following steps:

[0006] (1) Organize the experimental dataset, perform desensitization information processing and text data preprocessing on the dataset; the experimental dataset mainly includes the public datasets THUCnews, 20newsgroups and the constructed water conservancy document dataset.

[0007] (1.1) Stop word removal: First, construct a stop word set for the water conservancy field, add the existing stop word list to the stop word set for the water conservancy field, use the Jieba word segmentation tool to segment the input text data, query the constructed stop word set during the segmentation process, if the current word appears in the stop word set, filter it to complete the stop word removal;

[0008] (1.2) Remove punctuation marks: First, construct a punctuation mark set, add the existing punctuation mark table to the set, use the Jieba word segmentation tool to segment the input text data, query the constructed punctuation mark set during the segmentation process, if the current word appears in the punctuation mark set, filter it, and complete the punctuation mark cleaning.

[0009] (1.3) Spelling Correction: Spelling correction is performed using minimum edit distance, which refers to the minimum number of edits required to correct a misspelled word. Editing here includes three operations: insertion, deletion, and modification, and each edit can only change one letter. The correction process includes: identifying misspelled words; if a word is not in the existing vocabulary, it is considered an misspelled word; calculating the minimum edit distance for it and constructing a candidate edit list; editing here includes four operations: insertion, deletion, swapping, and replacement; through these four operations, all possible combinations are obtained to form the edit list; after enumerating and filling in the blanks in the edit list, only the correct words in the vocabulary are retained; the probability of the correct word is calculated using word probability; and the word with the highest word probability is selected as the result of spelling correction.

[0010] (2) Construct a topic attention model GAN-BiGRU Topic Attention Model based on topic mining and bidirectional adversarial neural network and bidirectional GRU, and train it using the training set in step (1);

[0011] The topic attention model GAN-BiGRU, which is based on subject mining and jointly uses bidirectional adversarial neural networks and bidirectional GRU, as described in step (2), mainly includes the following:

[0012] (2.1) Bidirectional Adversarial Neural Network Module: This module has two main functions. First, it is used to calculate the topic distribution of the input text. After the bidirectional GRU module outputs the text classification result, the classified text to be retrieved is retrieved through the topic distribution. Second, it uses the topic word vector as the Value in the Attention calculation of the bidirectional GRU module, and calculates the Attention weights with the text vector, as shown in the following formula:

[0013]

[0014] The Attention weights are calculated using Formula 1, where k is the initial linear mapping matrix in the attention mechanism, Q is the initial linear mapping matrix in the attention mechanism, dim is the dimension of the word embeddings in the input text, and V is the initial linear mapping matrix in the attention mechanism. This invention uses multiple topic vectors Vi as the Query input to the Attention layer. These Queries can assist the bidirectional GRU supervised model mentioned below in focusing on different aspects of the corpus from a global topic perspective. This avoids the potential information omission caused by traditional GRU networks primarily focusing on local contextual information.

[0015] (2.2) Bidirectional GRU Network Module: This module mainly has the following functions: taking the preprocessed text data in step (1) as input, embedding words into the text input, inputting the word vectors of the word embedding into the bidirectional GRU network, using the output of the bidirectional GRU network as Query and Key, calculating the Attention weights with the main distribution of the bidirectional adversarial neural network module, and finally outputting the classification information of the input text;

[0016] Bidirectional GRU is a supervised neural network model, while bidirectional adversarial neural network is an unsupervised neural topic model. In practice, the global topic vector learned in the neural topic model is used as the attention query input in the bidirectional GRU neural network, and the two models are jointly and synchronously learned through backpropagation.

[0017] (3) Test the topic diversity and topic coherence of the bidirectional adversarial neural network and bidirectional GRU joint topic attention model trained in step (2), and test the accuracy and recall of its downstream tasks to complete the model evaluation.

[0018] (3.1) Topic Diversity Test: The topic diversity after model training is reflected by statistically analyzing the proportion of non-repeating words in the topic vocabulary. The calculation formula is as follows:

[0019]

[0020] Where Set(topic_words) represents the set of topic words, topic_num represents the number of topics, and topic_per_word represents the number of words per topic; the topic diversity evaluation index can be calculated using this formula, and thus the topic diversity test can be completed.

[0021] (3.2) Topic Coherence Test: This indicator is used to evaluate whether words within the same topic have coherence and consistency. In this invention, NPMI and UCI values ​​are used as evaluation indicators. The calculation formulas for NPMI and UCI are as follows:

[0022]

[0023] Among them, w i Let w be the word vector of the i-th word. j Let P(w) be the word vector of the j-th word. i ,w j ) for w i and w j The joint probability distribution function, p(w) i ) and p(w j ) are w i and w j The marginal probability distribution function, where ε is the bias term;

[0024]

[0025] Among them, w i Let w be the word vector of the i-th word. j Let P(w) be the word vector of the j-th word. i ,w j ) for w i and w j The joint probability distribution function, p(w) i ) and p(w j ) are w i and w j The marginal probability distribution function, where N represents the vocabulary size;

[0026] (3.3) Accuracy: Accuracy = (TP + TN) / (P + N), representing the probability of correctly classifying a downstream classification task, where TP, TN, P, and N represent the metrics in the confusion matrix, i.e.:

[0027] The classification target has only two categories, namely positive examples and negative examples, which are:

[0028] True positives (TP): The number of positive samples correctly classified as positive, i.e., positive samples are predicted as positive;

[0029] False positives (FP): The number of positive samples that were incorrectly classified as positive, i.e., predicted as negative.

[0030] False negatives (FN): The number of instances that were incorrectly classified as negative, i.e., negative samples were predicted as positive.

[0031] True negatives (TN): The number of instances that are correctly classified as negative, i.e., negative samples are predicted as negative.

[0032] (3.4) Recall: The ratio of the number of positive samples correctly identified in the downstream task to the total number of positive samples in the test set. The calculation method is: Recall = TP / (TP + FN).

[0033] (4) Build a water conservancy field text retrieval system based on topic mining, embed a verified bidirectional adversarial neural network and bidirectional GRU joint topic attention model, and complete the relevant field text retrieval.

[0034] The water conservancy field text retrieval system based on topic mining described in step (4) mainly includes the following modules: data crawling module, index building module, and data retrieval module;

[0035] The data crawling module uses the Scrapy framework to automatically crawl the Ministry of Water Resources website using multiple threads and uses MongoDB for persistent storage.

[0036] The index building module will perform word segmentation on the crawled data text, filter it according to part of speech, then generate paragraph-topic probability distributions through topic model training of the corpus, and finally build the index using an ElasticSearch cluster.

[0037] The data retrieval module extracts topics based on paragraphs or keywords input by the user, then searches in the Elasticsearch cluster, and finally sorts them according to the Topic-BM25 topic relevance algorithm to provide the retrieval results.

[0038] A text retrieval system for the water resources field based on topic mining includes the following modules:

[0039] Data crawling module: The Scrapy framework is used to automatically crawl the website of the Ministry of Water Resources using multiple threads and the data is persisted using MongoDB;

[0040] Index building module: The crawled data text is segmented into words, then filtered according to part of speech, and then the corpus is trained with a topic model to generate the probability distribution of paragraphs-topics. Finally, the index is built using an ElasticSearch cluster.

[0041] Data retrieval module: Extracts topics based on paragraphs or keywords entered by the user, then retrieves them in the Elasticsearch cluster, and finally sorts them according to the Topic-BM25 topic relevance algorithm to provide the retrieval results.

[0042] A Topic-BM25 algorithm based on topic keyword weights is proposed. This algorithm not only considers the relevance of search terms but also integrates topic models to comprehensively consider their topic weights. The algorithm prioritizes displaying document paragraphs that are closely related to the search content topic and have high keyword relevance scores. The calculation formula is as follows:

[0043] score(t,D,Q)=wordScore(D,Q)*α+(1-α)*topicW(t,D)

[0044] Where wordScore(D,Q) represents the keyword relevance score, topicW(t,D) represents the topic word score, and α is the weighting coefficient of the relevance score. The calculation formulas for wordScore(D,Q) and topicW(t,D) are as follows:

[0045]

[0046]

[0047] Where, f(q) i D) is the word q i In document D, the TF value is: k1 is a constant representing word frequency saturation, generally between 1.2 and 2.0; b is a constant representing field length reduction, between 0 and 1; |D| is the text length; avg(dl) is the average text length; IDF(q) is the text length. i The inverse text frequency index (IF) is calculated as follows:

[0048]

[0049] Where docCount represents the total number of documents, and docFreq represents the number of documents containing the word t.

[0050] A computer storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for retrieving text in the water resources field based on topic mining.

[0051] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for retrieving water conservancy-related texts based on topic mining.

[0052] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0053] 1. This invention achieves significantly higher accuracy and recall than existing methods in topic diversity detection, topic coherence detection, and downstream classification tasks, providing a new solution for research in this related field;

[0054] 2. Based on the support of sufficient domain data reserves, this invention takes the lead in applying the network model formed by combining bidirectional adversarial neural networks and bidirectional GRUs to the field of water conservancy retrieval, thus contributing to the application of topic model retrieval in the field of water conservancy. Attached Figure Description

[0055] Figure 1 This is a flowchart of the present invention;

[0056] Figure 2 System architecture diagram;

[0057] Figure 3 This is a diagram of a network model formed by combining a bidirectional adversarial neural network and a bidirectional GRU. Detailed Implementation

[0058] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0059] like Figure 1 As shown, a method for retrieving text in the water resources field based on topic mining includes the following steps:

[0060] (1) Organize the experimental dataset, which mainly includes the public datasets THUCnews, 20newsgroups and the constructed water conservancy official document dataset. Perform desensitization information processing and text data preprocessing on the dataset.

[0061] The text preprocessing of the dataset in step (1) mainly includes the following steps:

[0062] (1.1) To remove stop words, first construct a stop word set for the water conservancy field, add the existing stop word list to the stop word set for the water conservancy field, use the Jieba word segmentation tool to segment the input text data, query the constructed stop word set during the word segmentation process, and filter out the current word if it appears in the stop word set to complete the removal of stop words;

[0063] (1.2) Remove punctuation marks. First, construct a punctuation mark set and add the existing punctuation mark table to the set. Use the Jieba word segmentation tool to segment the input text data. During the word segmentation process, query the constructed punctuation mark set. If the current word appears in the punctuation mark set, filter it to complete the punctuation mark cleaning.

[0064] (1.3) Spelling correction: Spelling correction is performed using minimum edit distance, which refers to the minimum number of edits required to correct a misspelled word. Editing here includes insertion, deletion, and modification operations, and each edit can only change one letter. The correction process includes: identifying misspelled words; if a word is not in the existing vocabulary, it is considered an error; calculating its minimum edit distance and constructing a candidate edit list; editing operations include insertion, deletion, swapping, and replacement operations; and obtaining all possible combinations through these four operations to form the edit list. After enumerating and filling in the edit list, only the correct words in the vocabulary are retained. The probability of the correct word is calculated using word probability, and the word with the highest probability is selected as the result of spelling correction.

[0065] (2) Construct a topic attention model GAN-BiGRU based on subject mining and bidirectional adversarial neural network and bidirectional GRU, and train it using the training set in step (1);

[0066] The construction of the GAN-BiGRU Topic Attention Model, which is a joint project of a bidirectional adversarial neural network and a bidirectional GRU based on subject mining, in step (2) mainly includes:

[0067] (2.1) Bidirectional Adversarial Neural Network Module: This module has two main functions. First, it is used to calculate the topic distribution of the input text. After the bidirectional GRU module outputs the text classification result, the module retrieves the classified text to be retrieved based on the topic distribution. Second, it uses the topic word vectors as the Value in the Attention calculation of the bidirectional GRU module, and calculates the Attention weights with the text vectors.

[0068] As mentioned in (21) above, this invention uses multiple topic vectors Vi as the Query input for the Attention layer. These Queries can assist the bidirectional GRU supervised model mentioned below in focusing on different aspects of information in the corpus from a global topic perspective. This avoids the potential information omission caused by traditional GRU networks mainly focusing on local contextual information.

[0069] (2.2) Bidirectional GRU Network Module: This module mainly has the following functions: taking the preprocessed text data in step (1) as input, performing word embedding on the text input, such as... Figure 3 As shown, the word vectors after word embedding are input into a bidirectional GRU network. The output of the bidirectional GRU network is used as the Query and Key, and the Attention weights are calculated with the main distribution of the bidirectional adversarial neural network module. Finally, the classification information of the input text is output.

[0070] Bidirectional GRU is a supervised neural network model, while bidirectional adversarial neural network is an unsupervised neural topic model. In practice, the global topic vector learned in the neural topic model is used as the attention query input in the bidirectional GRU neural network, and the two models are jointly and synchronously learned through backpropagation.

[0071] (3) Test the topic diversity and topic coherence of the bidirectional adversarial neural network and bidirectional GRU joint topic attention model trained in step (2) above, and test the accuracy and recall of its downstream tasks to complete the model evaluation.

[0072] The topic attention model jointly trained by the bidirectional adversarial neural network and bidirectional GRU in step (2) mentioned in step (3) is tested for topic diversity and topic coherence. At the same time, the accuracy and recall of its downstream tasks are tested. The model evaluation mainly includes:

[0073] (3.1) Topic diversity test: This indicator reflects the topic diversity after model training by statistically analyzing the proportion of non-repeating words in the topic vocabulary.

[0074] (3.2) Thematic coherence test: This indicator is used to evaluate whether words in the same topic have coherence and consistency. In this invention, NPMI value and UCI value will be used as evaluation indicators.

[0075] (3.3) Accuracy: Accuracy = (TP + TN) / (P + N), representing the probability of correctly classifying a downstream classification task. It is a key evaluation metric, and the higher the better. TP, TN, P, and N represent the metrics in the confusion matrix, i.e.:

[0076] The classification target has only two categories, labeled as positive and negative examples:

[0077] True positives (TP): The number of positive samples correctly classified as positive, i.e., positive samples are predicted as positive;

[0078] False positives (FP): The number of positive samples that were incorrectly classified as positive, i.e., predicted as negative.

[0079] False negatives (FN): The number of instances that were incorrectly classified as negative, i.e., negative samples were predicted as positive.

[0080] True negatives (TN): The number of instances that are correctly classified as negative, i.e., negative samples are predicted as negative.

[0081] (3.4) Recall: The ratio of the number of positive samples correctly identified in the downstream task to the total number of positive samples in the test set. The calculation method is: Recall = TP / (TP + FN).

[0082] (4) Build a water conservancy field text retrieval system based on topic mining, embed a verified bidirectional adversarial neural network and bidirectional GRU joint topic attention model, and complete the relevant field text retrieval;

[0083] like Figure 2 As shown, a text retrieval system for the water conservancy field based on topic mining mainly includes the following modules: a data crawling module, an index building module, and a data retrieval module.

[0084] Data crawling module: The Scrapy framework is used to automatically crawl the website of the Ministry of Water Resources using multiple threads and the data is persisted using MongoDB;

[0085] Index building module: The crawled data text is segmented into words, then filtered according to part of speech, and then the corpus is trained with a topic model to generate the probability distribution of paragraphs-topics. Finally, the index is built using an ElasticSearch cluster.

[0086] The data retrieval module extracts topics based on user-input paragraphs or keywords, retrieves them in the Elasticsearch cluster, and finally sorts them according to a topic relevance algorithm to provide the search results. The topic relevance algorithm is a Topic-BM25 algorithm based on topic word weights. This algorithm not only considers the relevance of search terms but also combines topic models to comprehensively consider their topic weights. The algorithm will prioritize displaying document paragraphs that are closely related to the search content topic and have a high comprehensive score of keyword relevance.

Claims

1. A method for retrieving text in the water resources field based on topic mining, characterized in that, Includes the following steps: (1) Organize the experimental dataset, perform desensitization processing and text data preprocessing on the dataset; (2) Construct a topic attention model GAN-BiGRU Topic Attention Model based on topic mining and bidirectional adversarial neural network and bidirectional GRU, and train it using the training set in step (1); (3) Test the topic diversity and topic coherence of the topic attention model jointly trained by the bidirectional adversarial neural network and bidirectional GRU in step (2), and test the accuracy and recall of its downstream tasks to complete the model evaluation; specifically: (3.1) Topic Diversity Test: The topic diversity after model training is reflected by statistically analyzing the proportion of non-repeating words in the topic vocabulary. The calculation formula is as follows: Where Set(topic_words) represents the set of topic words, topic_num represents the number of topics, and topic_per_word represents the number of words for each topic; the topic diversity evaluation index can be calculated using this formula, and thus the topic diversity test can be completed. (3.2) Topic Coherence Test: This indicator is used to evaluate whether words within the same topic have coherence and consistency. In this invention, NPMI and UCI values ​​are used as evaluation indicators. The calculation formulas for NPMI and UCI are as follows: Among them, w i Let w be the word vector of the i-th word. j Let P(w) be the word vector of the j-th word. i ,w j ) for w i The joint probability distribution function of w, p(w) i ) and p(w j ) are w i and w j The marginal probability distribution function, where ε is the bias term; Among them, w i Let w be the word vector of the i-th word. j Let P(w) be the word vector of the j-th word. i ,w j ) for w i The joint probability distribution function of w, p(w) i ) and p(w j ) are w i and w j The marginal probability distribution function, where N represents the vocabulary size; (3.3) Accuracy: Accuracy = (TP + TN) / (P + N), representing the probability of correctly classifying a downstream classification task, where TP, TN, P, and N represent the metrics in the confusion matrix, i.e.: The classification target has only two categories, namely positive examples and negative examples, which are: True positives (TP): The number of positive samples correctly classified as positive, i.e., positive samples are predicted as positive; False positives (FP): The number of positive samples that were incorrectly classified as positive, i.e., predicted as negative. False negatives (FN): The number of instances that were incorrectly classified as negative, i.e., negative samples were predicted as positive. True negatives (TN): The number of instances that are correctly classified as negative, i.e., negative samples are predicted as negative; (3.4) Recall: The ratio of the number of positive samples correctly identified in the downstream task to the total number of positive samples in the test set. The calculation method is: Recall = TP / (TP + FN); (4) Build a water conservancy field text retrieval system based on topic mining, embed a verified bidirectional adversarial neural network and bidirectional GRU joint topic attention model, and complete the relevant field text retrieval.

2. The method for retrieving text in the water conservancy field based on topic mining according to claim 1, characterized in that, The experimental dataset mentioned in step (1) mainly includes the public datasets THUCnews, 20newsgroups, and the constructed water conservancy document dataset.

3. The method for retrieving text in the water conservancy field based on topic mining according to claim 1, characterized in that, The specific steps (1) are as follows: (1.1) Stop word removal: First, construct a stop word set for the water conservancy field, add the existing stop word list to the stop word set for the water conservancy field, use the Jieba word segmentation tool to segment the input text data, query the constructed stop word set during the segmentation process, if the current word appears in the stop word set, filter it to complete the stop word removal; (1.2) Remove punctuation marks: First, construct a punctuation mark set, add the existing punctuation mark table to the set, use the Jieba word segmentation tool to segment the input text data, query the constructed punctuation mark set during the segmentation process, if the current word appears in the punctuation mark set, filter it, and complete the punctuation mark cleaning. (1.3) Spelling correction: Spelling correction is completed using the minimum edit distance, which refers to the minimum number of edits required to correct a misspelled word. Editing here includes three operations: insertion, deletion, and modification, and each edit can only change one letter. The error correction process includes: identifying misspelled words; if a word is not in the existing vocabulary, it is considered an error word; calculating the minimum edit distance for it and constructing a candidate edit list; the edit operations here include four types: insertion, deletion, swapping, and replacement; through these four operations, all possible combinations are obtained to form the edit list; after enumerating and filling in the blanks in the edit list, only the correct words in the vocabulary are retained; the probability of the correct word is calculated using word probability; and the word with the highest word probability is selected as the result of the spell correction.

4. The method for retrieving text in the water conservancy field based on topic mining according to claim 1, characterized in that, The topic attention model GAN-BiGRU, which is based on subject mining and jointly uses bidirectional adversarial neural networks and bidirectional GRU, as described in step (2), mainly includes the following: (2.1) Bidirectional Adversarial Neural Network Module: This module has two main functions. First, it is used to calculate the topic distribution of the input text. After the bidirectional GRU module outputs the text classification result, the classified text to be retrieved is retrieved through the topic distribution. Second, it uses the topic word vector as the Value in the Attention calculation of the bidirectional GRU module, and calculates the Attention weights with the text vector, as shown in the following formula: The calculation formula for the attention weight is Formula 1, where k is the initial linear mapping matrix in the attention mechanism, Q is the initial linear mapping matrix in the attention mechanism, dim is the dimension of the word embedding in the input text, and V is the initial linear mapping matrix in the attention mechanism. This invention uses multiple topic vectors Vi as the Query input of the Attention layer. (2.2) Bidirectional GRU Network Module: This module mainly has the following functions: taking the preprocessed text data in step (1) as input, embedding words into the text input, inputting the word vectors of the word embedding into the bidirectional GRU network, using the output of the bidirectional GRU network as Query and Key, calculating the Attention weights with the main distribution of the bidirectional adversarial neural network module, and finally outputting the classification information of the input text; Bidirectional GRU is a supervised neural network model, while bidirectional adversarial neural network is an unsupervised neural topic model. In practice, the global topic vector learned in the neural topic model is used as the attention query input in the bidirectional GRU neural network, and the two models are jointly and synchronously learned through backpropagation.

5. The method for retrieving text in the water conservancy field based on topic mining according to claim 1, characterized in that, The water conservancy field text retrieval system based on topic mining described in step (4) mainly includes the following modules: data crawling module, index building module and data retrieval module; The data crawling module uses the Scrapy framework to automatically crawl the Ministry of Water Resources website using multiple threads and uses MongoDB for persistent storage. The index building module will perform word segmentation on the crawled data text, filter it according to part of speech, then generate the paragraph-topic probability distribution through topic model training of the corpus, and finally build the index using an ElasticSearch cluster. The data retrieval module extracts topics based on paragraphs or keywords input by the user, then searches in the Elasticsearch cluster, and finally sorts them according to the Topic-BM25 topic relevance algorithm to provide the retrieval results.

6. A text retrieval system for the water conservancy field based on topic mining, characterized in that, Includes the following modules: Data crawling module: The Scrapy framework is used to automatically crawl the website of the Ministry of Water Resources using multiple threads and the data is persisted using MongoDB; Index building module: It processes text data by word segmentation, filters it according to part of speech, generates paragraph-topic probability distributions by training a topic model on the corpus, and finally builds the index using an ElasticSearch cluster. Data retrieval module: Extracts topics based on user-input paragraphs or keywords, retrieves them in the Elasticsearch cluster, and finally sorts them according to the Topic-BM25 topic relevance algorithm to provide search results. This algorithm not only considers the relevance of search terms but also incorporates a topic model to comprehensively consider topic weights. The algorithm prioritizes displaying document paragraphs that are closely related to the search content's topic and have high keyword relevance scores. The calculation formula is as follows: score(t,D,Q)=wordScore(D,Q)*α+(1-α)*topicW(t,D) Where wordScore(D,Q) represents the keyword relevance score, topicW(t,D) represents the topic word score, and α is the weighting coefficient of the relevance score. The calculation formulas for wordScore(D,Q) and topicW(t,D) are as follows: Where, f(q) i D) is the word q i In document D, the TF value is: k1 is a constant representing word frequency saturation, generally between 1.2 and 2.0; b is a constant representing field length reduction, between 0 and 1; |D| is the text length; avg(dl) is the average text length; IDF(q) is the text length. i The inverse text frequency index (IF) is calculated as follows: Where docCount represents the total number of documents, and docFreq represents the number of documents containing the word t.

7. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements a method for retrieving water conservancy-related texts based on topic mining as described in any one of claims 1-5.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a method for retrieving water conservancy-related texts based on topic mining as described in any one of claims 1-5.

Images