LDA topic model identification method

Abstract

The invention relates to an LDA topic model recognition method which comprises the following steps: S1, collecting high-frequency search keywords of archives by utilizing a Python library, preprocessing the collected data, including removing useless punctuations and special symbols, only extracting Chinese and performing text word segmentation, and meanwhile, deepening to the Internet or a specific database by adopting a crawler technology, so as to obtain an LDA topic model; according to the method, richer and more comprehensive archive data are collected, a natural language processing (NLP) technology is introduced, deeper semantic understanding and information extraction are performed, non-key vocabularies are filtered by adopting part-of-speech tagging, and the data quality and the analysis precision are improved. Based on an LDA topic model, various sampling algorithms are adopted to allocate categories for training data, after convergence is achieved, the score of each category of to-be-classified books is counted, the category of to-be-classified archives is obtained accordingly, and aiming at the defects that a traditional archive format is not standard, naming is not standard, and archive sharing access is difficult in organizations at all levels, the classification efficiency is improved. The improvement is obvious.

Description

Technical Field

[0001] The present invention relates to the technical field of topic model recognition, and in particular to an LDA topic model recognition method. Background Art

[0002] With the continuous advancement of information technology, the construction of digital archives has been widely promoted and applied around the world. The construction of digital archives not only involves converting paper archives into electronic formats, but more importantly, it involves establishing an effective information resource management system to achieve efficient management and convenient sharing of archives. This system is usually network-based, distributed, and has an object-oriented network structure model, covering the entire process from the creation, storage, retrieval, and sharing of a single electronic archive.

[0003] However, although such a system greatly facilitates the storage, access and sharing of information, archive management still faces many challenges in actual operation. One of them is the inconsistency in the format and naming of various electronic archives, which makes it difficult to integrate and quickly retrieve information resources. In addition, organizations of different levels or types may find it difficult to effectively share and access archives due to differences in technology, policies and other factors.

[0004] Based on this, those skilled in the art have proposed an LDA topic model identification method. Summary of the invention

[0005] In view of the above problems existing in the prior art, the main purpose of the present invention is to provide an LDA topic model identification method.

[0006] The technical solution of the present invention is as follows: a LDA topic model identification method, comprising the following steps:

[0007] S1. Use Python library to collect high-frequency search keywords of archives and pre-process the collected data, including removing useless punctuation and special symbols, extracting only Chinese characters, and performing text segmentation. At the same time, crawler technology is used to go deep into the Internet or specific databases to collect richer and more comprehensive archive data. Natural language processing (NLP) technology is introduced to conduct deeper semantic understanding and information extraction. Part-of-speech tagging is used to filter non-key words to improve the quality of data and the accuracy of analysis. Data deduplication technology is introduced to ensure the uniqueness and effectiveness of analysis.

[0008] S2. Use Jieba word segmentation to optimize performance, introduce domain-specific vocabulary libraries and custom dictionaries, use Jieba's Chinese word segmentation library, combine context to segment text, and improve the accuracy and professionalism of word segmentation. At the same time, develop a real-time update mechanism to dynamically adjust the word segmentation strategy as new words emerge and the frequency of use changes. In addition, explore the combined use of multiple word segmentation algorithms, compare the advantages and disadvantages of each algorithm, and select the word segmentation scheme that best suits the current text collection;

[0009] S3. Based on the Term Frequency-Inverse Document Frequency (TF-IDF) model, the text is converted into a bag-of-words model, and the TF-IDF matrix is ​​used to create a vocabulary. The accuracy of topic template mining with characteristic words is improved through the two indicators of term frequency and inverse document frequency. In the process of building the bag-of-words model, the language model and deep learning technology are introduced to further capture and understand the deep semantic relationship in the text and enhance the model's expression ability. In addition, the TF-IDF model is optimized and the influence of different parameter settings on the model effect is explored to achieve the best performance.

[0010] S4. Use the LDA (Latent Dirichlet Allocation) topic model to extract topics, and determine the optimal number of topics by calculating the PV (perplexity-variance) indicator, so as to determine the most appropriate number of topics that can reflect the topic structure of the document collection. When using the LDA topic model to extract topics, combine other variants of the topic model (such as the dynamic topic model DTM, structured topic model STM) to explore the evolution of topics over time or the potential structural relationship between topics. When determining the optimal number of topics, introduce more mathematical indicators and statistical tests, and make a comprehensive judgment through multiple indicators to determine the most reasonable number of topics, so as to improve the accuracy and reliability of topic extraction.

[0011] S5. Based on the determined optimal number of topics, an LDA model is constructed and trained. This process involves learning topic distribution from documents and word distribution corresponding to topics. The training of the model aims to capture the potential topic structure in text data. In the LDA model training process, cross-validation and grid search mechanisms are introduced to optimize the model's hyperparameter selection. At the same time, parallel computing technology is explored to speed up the model training process. In addition, external knowledge bases or pre-trained models are integrated to deepen the model's understanding and adaptability to specific fields, so as to improve the training effect.

[0012] S6. Based on the training results of the LDA model, the archives are automatically classified and archived by subject. After the trained LDA model completes the automatic classification, a manual verification and feedback mechanism is further introduced to continuously optimize the classification model. At the same time, the machine learning algorithm (such as support vector machine SVM, neural network) is explored for multi-level and refined classification to provide a more orderly and easy-to-retrieve archive structure.

[0013] S7. Evaluate the performance of the LDA model through perplexity and topic consistency indicators to verify the model effect and ensure that the model can effectively predict the topic distribution of new documents. In addition to using perplexity and topic consistency indicators, introduce other evaluation indicators (such as precision, recall, F1 value) to conduct a comprehensive evaluation of model performance. At the same time, establish a visual display system for model performance evaluation to intuitively display the evaluation results, so as to quickly identify the advantages and disadvantages of the model.

[0014] S8. Display the classification results of the LDA model, including topic keyword distribution and cluster visualization. Use visualization tools to create interactive dashboards to display advanced analysis views such as topic evolution and keyword co-occurrence networks. Customize the display results for different user groups to improve the accessibility and ease of use of information.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are:

[0016] The present invention improves the automatic identification and archiving of digital archives in the government cloud through a word frequency training model to improve the integrity of digital archive management and the security of cross-departmental access. The automatic classification system is trained using the LDA topic model, including a classification system for digital archives through keyword categories, selecting archives of established categories as training archive contents, and the subject labels of all training archives constitute the total set of archive labels, and assigning a sequence to each label in the total set of archives, constructing and training a multinomial distribution model. In addition, this solution is based on the LDA topic model and uses a variety of sampling algorithms to assign categories to training data. When convergence is reached, the scores of each category to which the books to be classified belong are counted, and the category to which the archives to be classified belong is obtained accordingly. It has significant improvements in view of the defects of traditional archive formats that are not standardized, naming that is not standardized, and difficulties in archive sharing and access in organizations at all levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the overall steps of the present invention;

[0018] Figure 2 is a detailed flow chart of step S1 of the present invention;

[0019] Figure 3 is a detailed flow chart of step S2 of the present invention;

[0020] Figure 4 is a detailed flow chart of step S3 of the present invention;

[0021] Figure 5 is a detailed flow chart of step S4 of the present invention;

[0022] Figure 6 is a detailed flow chart of step S5 of the present invention;

[0023] Figure 7 is a detailed flow chart of step S6 of the present invention;

[0024] Figure 8 is a detailed flow chart of step S7 of the present invention;

[0025] Fig. 9 is a detailed flow chart of step S8 of the present invention;

[0026] Fig.10 It is a diagram of the generation process of the LDA topic model of the present invention;

[0027] Fig.11 Identifying key framework graphs for the topic model of the present invention;

[0028] Fig.12 It is the PV topic number-perplexity-variance curve diagram of the present invention;

[0029] Fig.13 This is a clustering visualization display diagram of the co-word matrix of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0032] Example

[0033] like Figure 1-Figure 13 As shown, the present invention provides an LDA topic model identification method, which includes the following steps:

[0034] S1. Use Python libraries to collect the high-frequency search keywords of the archives, and preprocess the collected data, including removing useless punctuation and special symbols, only extracting Chinese, and performing text word segmentation. At the same time, use web crawler technology to deeply explore the Internet or specific databases to collect more abundant and comprehensive archive data, introduce natural language processing (NLP) technology to perform deeper semantic understanding and information extraction, use part-of-speech tagging to filter out non-keyword vocabulary, improve the quality of the data and the accuracy of the analysis, and introduce data deduplication technology to ensure the uniqueness and effectiveness of the analysis;

[0035] The step S1 can be further divided into:

[0036] S11. Use the Requests and BeautifulSoup libraries of Python to crawl data from the network or specific databases. This step includes writing accurate web requests, parsing the returned HTML or JSON data to obtain the required information. The key to this step is to locate and correctly extract the keywords or phrases in the documents, archives or web pages;

[0037] S12. After obtaining the data, perform preliminary data cleaning and preprocessing, including removing non-content parts such as HTML tags, JavaScript code, CSS styles, etc., then剔除 some useless punctuation marks and special characters, and extract the Chinese or key words for analysis;

[0038] S13. Use the Jieba word segmentation library to perform Chinese word segmentation on the cleaned text. Jieba word segmentation can automatically adjust the dictionary according to the word frequency and accurately separate the effective vocabulary in the article;

[0039] S14. By comparing with the custom stop word dictionary, filter out the words that have no practical meaning in the analysis, such as "of", "is", "in", etc., so as to reduce the noise in the subsequent processing and improve the attention and accuracy of the data analysis;

[0040] S15. Use natural language processing technology to further expand and understand the semantics of the word segmentation results, including part-of-speech tagging and semantic understanding, and can perform customized processing on the keywords in specific fields to improve the professionalism and accuracy of the word segmentation;

[0041] S16. After all preprocessing work is completed, the data is deduplicated to ensure the uniqueness and validity of the analyzed data. In addition, the processed data needs to be reasonably integrated and stored to facilitate subsequent data analysis and mining. This step can effectively improve the quality and efficiency of archival data collection, processing and analysis. First, the advanced crawler technology combined with the Requests and BeautifulSoup libraries can accurately collect rich archival data from the Internet or specific databases. Subsequently, through automated data preprocessing steps, including removing irrelevant characters, Chinese word segmentation, stop word filtering, etc., useful information can be accurately extracted, data noise can be reduced, and the accuracy of data analysis can be improved. The introduction of natural language processing (NLP) technology not only deepens the semantic understanding of text content, but also further screens out key words through methods such as part-of-speech tagging, enhancing the professionalism of data processing. In addition, the application of data deduplication technology ensures the uniqueness and validity of the analysis results and ensures the accuracy of data analysis.

[0042] In this step, the word segmentation function is as follows:

[0043] Define the word cutting function chinese_word_cut(mytext):

[0044] (1) Remove useless characters to extract Chinese characters

[0045] (2) Text preprocessing: remove useless characters, such as punctuation marks and special symbols, and extract Chinese characters

[0046] (3) Perform text segmentation

[0047] (4) Read the stop dictionary.txt, read the text line by line, and filter out the useless words that match the dictionary

[0048] (5) Save the segmented words in the sisoco.csv file.

[0049] (6) Create a list to store segmented words

[0050] def chinese_word_cut(mytext):

[0051]

[0052]

[0053] S2. Use Jieba word segmentation to optimize performance, introduce domain-specific vocabulary libraries and custom dictionaries, use Jieba's Chinese word segmentation library, combine context to segment text, and improve the accuracy and professionalism of word segmentation. At the same time, develop a real-time update mechanism to dynamically adjust the word segmentation strategy as new words emerge and the frequency of use changes. In addition, explore the combined use of multiple word segmentation algorithms, compare the advantages and disadvantages of each algorithm, and select the word segmentation scheme that best suits the current text collection;

[0054] The step S2 can be subdivided into:

[0055] S21. In the initial stage of large-scale text data processing, Jieba is first used to perform basic text segmentation operations. Since Jieba supports user-defined dictionaries, corresponding professional vocabulary libraries are prepared in advance according to the specific fields of archives, including government affairs, education, medical care, etc. By importing these professional vocabulary libraries into the Jieba word segmenter, the recognition rate of professional terms can be significantly improved, thereby ensuring the accuracy and professionalism of word segmentation to a certain extent;

[0056] S22. As time goes by and language develops, new words will continue to appear in the language. Therefore, it is necessary to develop a dynamic update mechanism to regularly capture new words and their usage frequency from the Internet or industry literature and update them to the dictionary of the word segmentation system. An automated vocabulary update mechanism can be implemented through API interfaces or crawler technology, so that the word segmentation strategy can respond to actual changes in language use in a timely manner and maintain word segmentation accuracy and timeliness.

[0057] S23. In order to further improve the accuracy of word segmentation, the meaning of words in specific contexts must be considered. For example, "apple" may mean both fruit and a technology company. By analyzing the contextual information before and after the word, the specific meaning of the word can be more accurately determined and a more accurate word segmentation decision can be made. NLP technologies such as part-of-speech tagging and dependency syntax analysis can help achieve this.

[0058] S24. A single word segmentation algorithm is difficult to handle all types of text data. Therefore, we explored the combination of Jieba with other word segmentation tools or algorithms (such as HMM, CRF and other models) to achieve better word segmentation results. Through the preset test set, we compared the performance of different word segmentation algorithms or algorithm combinations in terms of accuracy and efficiency, and selected the optimal word segmentation solution to apply to the project.

[0059] S25. After word segmentation, it is an important step to improve the quality of word segmentation to reasonably filter stop words (such as auxiliary words, conjunctions, etc.). Develop an automated stop word processing mechanism, dynamically adjust and optimize the stop word library in combination with the characteristics of the specific field of archives, and use the generated stop word library to filter the word segmentation results, excluding those words that have no actual semantic contribution, so as to improve the accuracy of subsequent text processing and analysis;

[0060] S26. In order to cope with the processing needs of large-scale text data, it is also necessary to optimize the performance of word segmentation operations from a technical level, and use technical means such as multi-threading and parallel processing to improve the processing speed of word segmentation. At the same time, the data structure generated after word segmentation is optimized to ensure efficient access and processing of data in subsequent processing. Using Jieba word segmentation to add professional vocabulary libraries and custom dictionaries not only improves the accuracy and professionalism of word segmentation, but also adapts to the continuous changes and development of language. The dynamic update mechanism can capture new vocabulary in a timely manner and update it to the word segmentation system to maintain the timeliness and accuracy of the word segmentation strategy. In addition, exploring the combined use and optimization of different word segmentation algorithms enables analysis tools to better cope with complex and diverse text data. After achieving these optimizations, the processing speed and quality of archival information can be effectively improved, thereby improving the efficiency and service quality of the entire archive management system and meeting the needs of modern digital archive management.

[0061] S3. Based on the Term Frequency-Inverse Document Frequency (TF-IDF) model, the text is converted into a bag-of-words model, and the TF-IDF matrix is ​​used to create a vocabulary. The accuracy of topic template mining with characteristic words is improved through the two indicators of term frequency and inverse document frequency. In the process of building the bag-of-words model, the language model and deep learning technology are introduced to further capture and understand the deep semantic relationship in the text and enhance the model's expression ability. In addition, the TF-IDF model is optimized and the influence of different parameter settings on the model effect is explored to achieve the best performance.

[0062] The step S3 can be subdivided into:

[0063] S31. In the initial stage, the TF-IDF model is used to convert the text into a bag-of-words model. By considering the statistics of word frequency and inverse document frequency, the recognition of important words in the document is strengthened. The model is built using TfidfVectorizer from the scikit-learn library, and the model performance is optimized by adjusting parameters (such as max_df and min_df, etc.), and common and rare words in the text are eliminated to highlight the distinguishing keywords.

[0064] S32. Combining word embedding and deep learning techniques (such as Word2Vec or BERT) into the traditional TF-IDF model can supplement the semantic information lost by TF-IDF due to ignoring word order and context, and improve the model's ability to capture the deep semantics of the text;

[0065] S33. Through experiments and verification of different parameter settings, find the optimal parameter combination, and apply techniques such as cross-validation to evaluate the effect of the model under different parameter configurations to optimize the final text processing performance;

[0066] S34. Customize the stop word list for specific data sets and text content. Considering the particularity of the archives that may contain professional terms and common words, appropriately adjust the stop word list to suppress or emphasize the weight of certain words, thereby improving the overall analysis accuracy and effectiveness;

[0067] S35. Use the LDA model to automatically identify and extract topics from the document set, determine the optimal number of topics by calculating the model perplexity and variance index (PV index), optimize the model settings through statistical and visualization methods, and verify the stability and accuracy of the model through topic consistency testing;

[0068] S36. After the model parameters are determined, the final model performance evaluation is carried out to ensure that the model can accurately classify and archive files, and the generality and practicality of the model are tested through actual application scenarios. The final adjustment and optimization are carried out according to the feedback. Through this step, the text analysis model is ensured to show good stability and efficiency not only during the research period but also in actual applications. First, the TF-IDF model is used to convert the text to the bag-of-words mode and the word frequency and inverse document frequency are combined to enhance the recognition of keywords. Further, the fusion of word embedding and deep learning technologies such as Word2Vec or BERT enhances the model's ability to recognize and understand the deep semantics of the text, thereby improving the expression and analysis performance of the model. Through parameter optimization and cross-validation, the best parameter combination is found to ensure that the model can achieve optimal performance on different text sets. In addition, the stop word list is customized to meet the needs of specific data sets and enhance the model's ability to handle professional terms. The LDA model is used to automatically identify topics, and the number of topics is optimized through the PV indicator to ensure that the extracted topics are both accurate and representative. This series of measures will ultimately improve the stability and practicality of text analysis, making the model not only effective in the research stage but also play an important role in practical applications, enhancing the overall performance and efficiency of document management and information retrieval.

[0069] Among them, the parameter calculation of TF-IDF is shown in the formula:

[0070] TF-IDF(t,d)=w(t,d)*IDF(T)

[0071] Convert text to bag-of-words mode;

[0072] vectorizer.fit_transform(file_object2) gets a sparse matrix, which can be used to obtain word frequency;

[0073] Create a vocabulary using the TF-IDF matrix and return a list containing the names of all the extracted feature words.

[0074] #Convert the text into bag-of-words mode

[0075] vectorizer=TfidfVectorizer()

[0076] tfidf_matrix=vectorizer.fit_transform(file_object2)

[0077] #Create a vocabulary using the TF-IDF matrix

[0078] dictionary=corpora.Dictionary([vectorizer.get_feature_names_out()])

[0079] corpus=[dictionary.doc2bow(text)for text in[vectorizer.get_feature_names_out()]]

[0080] #data_set=[]

[0081] #corpus=data_set['content_cutted']

[0082] #dictionary=corpora.Dictionary(data_set)#Build document-term matrix

[0083] #corpus=[dictionary.doc2bow(text)for text in data_set].

[0084] Calculate the PV index and determine the vocabulary corresponding to the minimum PV value as the optimal number of topics based on the calculation results.

[0085] There are two common indicators for evaluating the performance of LDA models, Perplexity and Conhence. These two indicators are particularly important for determining the optimal number of topics in indicator extraction. The Perplexity indicator is used to measure the model's ability to predict new documents. This indicator focuses on the generalization ability of the model. When there is a large number of extracted topics and the similarity between topics is too high, the Perplexity-Var (PV) indicator is proposed. It proposes to use auxiliary topic variance and composite perplexity to determine the optimal number of topics based on the perplexity calculation. The calculation formula is as follows:

[0086]

[0087] Where D represents the dataset, P(D) represents the perplexity of the dataset, and the smaller P is, the better the generalization ability of LDA is; T is the topic of the LDA calculation of the dataset, and Var(T) is the variance of the dataset. The larger the topic variance is, the better the effect of LDA topic calculation is. Formula 1 shows that the LDA topic model recognition effect is better when the PV index is the smallest.

[0088] In practical applications, the perplexity P(D) indicator is used to optimize the performance of the model. P(D) calculation is to predict the i-th word (W i The conditional probability of ) is calculated as follows:

[0089] P(D)=P(W1,W2,…,W i )=p(W1)P(W2|W1)…P(W k |W1,W2,…,W i-1 )

[0090] Variance of Topics is an important indicator used to measure the degree of dispersion between topics. The calculation formula of topic variance is as follows:

[0091]

[0092] In the above formula, ψ represents the mean of the topic-word probability distribution, KL is the topic T i and T ψ The larger the topic variance, the higher the distinction between topics, so the optimal number of topics can be determined.

[0093] Function code explanation: According to the perplexity-variance (PV) calculation in the LDA model, the optimal number of topics is determined. First, the lda function is defined, where the prior parameters α (alpha) = 0.05 and β (eta) are set to the default values. The number of model iterations is 100. The data obtained from the program running results is used to draw the PV topic number-perplexity-variance curve. The results are shown in Fig.12 , the lowest perplexity-variance value of 18 is selected as the optimal topic. At the same time, LDA is an unsupervised machine learning clustering method. The results are shown in Fig.13 As shown in the figure, the clustering visualization of the co-word matrix formed by high-frequency words, the topic keywords are the top 20 words in frequency by default; Fig.13 The cluster distribution of the analyzed words is shown in the figure. The cluster is set to 6 categories. The size of the bubble represents the frequency of the word, and the color represents the cluster category. The specific topic category-topic keyword distribution is as follows:

[0094]

[0095]

[0096]

[0097] S4. Use the LDA (Latent Dirichlet Allocation) topic model to extract topics, and determine the optimal number of topics by calculating the PV (perplexity-variance) indicator, so as to determine the most appropriate number of topics that can reflect the topic structure of the document collection. When using the LDA topic model to extract topics, combine other variants of the topic model (such as the dynamic topic model DTM, structured topic model STM) to explore the evolution of topics over time or the potential structural relationship between topics. When determining the optimal number of topics, introduce more mathematical indicators and statistical tests, and make a comprehensive judgment through multiple indicators to determine the most reasonable number of topics, so as to improve the accuracy and reliability of topic extraction.

[0098] The step S4 can be refined as follows:

[0099] S41. Based on the LDA model, consider introducing the dynamic topic model (DTM) and the structured topic model (STM) to capture the dynamic characteristics of the document collection evolving over time or explore the potential structural relationship between topics. Preliminary model parameters are set, such as the number of topics and the number of iterations. These parameters will be further optimized in subsequent steps.

[0100] S42. Preprocess the data according to the requirements of the LDA model, including text cleaning (removing stop words, punctuation marks, etc.), word segmentation, and building word bags. Use advanced text processing techniques, such as TF-IDF conversion, to reduce the impact of high-frequency non-topic words and enhance the recognition effect of the model.

[0101] S43. Use LDA and its variant models to extract the topic of the document, and conduct preliminary topic interpretation to evaluate the rationality and accuracy of the topic. This step can be performed using graphical tools such as PyLDAVis.

[0102] S44. Combine the perplexity variance (PV) indicator to adjust the number of topics and find the balance between minimizing perplexity and maximizing topic variance. In addition, consider introducing other mathematical indices such as document consistency test to evaluate the discrimination and consistency of topics.

[0103] S45. Compare the results under different models and settings, not only limited to PV indicators, but also including indicators such as topic consistency and document uniqueness. Combine the results of multiple indicators and use statistical methods to determine the optimal topic model parameters;

[0104] S46. In specific application scenarios, such as digitalization of archives management and government information systems, the optimized model is deployed for actual application testing, and the model parameters are further adjusted based on usage feedback. The theoretical design and practical performance of the model are evaluated through the comprehensive performance of actual scenarios. The introduction of the LDA model and its variants such as the dynamic topic model (DTM) and the structured topic model (STM) helps to accurately capture the characteristics of document evolution over time and explore the potential structural relationship between topics. The accuracy of the model is enhanced by reducing the interference of high-frequency non-topic words through advanced data preprocessing and the use of technologies such as TF-IDF. The determination of the number of topics is optimized by using PV indicators and other mathematical indicators, so that the model can better reflect the topic structure of the document under the premise of minimizing perplexity and maximizing topic variance. In addition, the comprehensive evaluation of multiple indicators can provide more scientific basis for determining the optimal model parameters. Finally, by testing the optimized model in specific application scenarios, it can ensure that the theoretical design of the model is fully integrated with the actual application performance, effectively improving the work efficiency and quality in fields such as digitalization of archives management and government information systems.

[0105] S5. Based on the determined optimal number of topics, an LDA model is constructed and trained. This process involves learning topic distribution from documents and word distribution corresponding to topics. The training of the model aims to capture the potential topic structure in text data. In the LDA model training process, cross-validation and grid search mechanisms are introduced to optimize the model's hyperparameter selection. At the same time, parallel computing technology is explored to speed up the model training process. In addition, external knowledge bases or pre-trained models are integrated to deepen the model's understanding and adaptability to specific fields, so as to improve the training effect.

[0106] The step S5 can be refined as follows:

[0107] S51. Before starting to build the LDA model, you first need to determine the optimal number of topics, which can be achieved by using the aforementioned PV indicator and other methods. At the same time, prepare the data, including preprocessing steps such as data cleaning, word segmentation, and word bag construction to ensure the quality and integrity of the data;

[0108] S52. After determining the optimal number of topics, start building the LDA model and set the model's hyperparameters, such as the number of iterations, α and β values, etc. The selection of these parameters will directly affect the performance and effect of the model, which needs to be adjusted and optimized according to the specific situation;

[0109] S53, introduce cross-validation technology and grid search mechanism, through repeated training of the model and evaluation on the validation set, select the best hyperparameter combination, so as to avoid overfitting and improve the generalization ability of the model, so that it performs better on new data;

[0110] S54. Consider using parallel computing technology to accelerate the model training process, especially when processing large-scale data, which can significantly shorten the training time, including using multi-core CPU parallel processing or GPU acceleration to improve training efficiency;

[0111] S55. Integrate external knowledge bases or pre-trained models into LDA model training to improve the model's understanding and adaptability to specific fields, including introducing domain-specific vocabulary libraries, terminology dictionaries, or using pre-trained word vector models, thereby improving the model's expressiveness and training results;

[0112] S56. After the model training is completed, the model is evaluated and the results are verified, including checking the performance of the model on the training set and the test set, and verifying the rationality of the topic distribution and keyword distribution. The model is comprehensively evaluated through a variety of evaluation indicators and visualization tools to ensure that its performance and effect meet the expected requirements. By determining the optimal number of topics and adopting appropriate preprocessing measures, the quality of the data and the basis of model training are guaranteed. Subsequently, by setting reasonable model hyperparameters and applying cross-validation and grid search techniques, the model performance can be optimized, overfitting can be avoided, and the generalization ability of the model on unknown data can be enhanced. The use of parallel computing technology, especially when processing big data, can significantly improve the speed and efficiency of model training. In addition, the integration of external knowledge bases and pre-trained models not only enhances the model's understanding ability, but also improves its adaptability to specific fields, making the training effect more accurate and comprehensive. Finally, through comprehensive model evaluation and result verification, it is ensured that the performance and effect of the model meet the expected goals, so as to achieve efficient and high-quality document topic extraction and classification. This method not only improves the practicality and adaptability of the model, but also provides strong technical support for subsequent research and application in related fields.

[0113] S6. Based on the training results of the LDA model, the archives are automatically classified and archived by subject. After the trained LDA model completes the automatic classification, a manual verification and feedback mechanism is further introduced to continuously optimize the classification model. At the same time, the machine learning algorithm (such as support vector machine SVM, neural network) is explored for multi-level and refined classification to provide a more orderly and easy-to-retrieve archive structure.

[0114] The step S6 can be refined as follows:

[0115] S61. Based on the trained LDA model, the archives are automatically classified and filed according to themes, and each document is mapped to the most relevant theme, thereby achieving automatic classification. After the classification is completed, preliminary manual verification is performed to ensure the accuracy and completeness of the classification results;

[0116] S62. Introduce a manual verification and feedback mechanism after classification, feed the manual verification results back to the model to optimize the classification model, and continuously improve and optimize the LDA model by collecting feedback information from manual verification to improve its classification accuracy and robustness;

[0117] S63. Explore the use of machine learning algorithms for multi-level and refined classification. In addition to the LDA model, consider introducing support vector machine (SVM), neural network and other machine learning algorithms to build more complex and accurate classification models. These algorithms can further improve the accuracy and effect of classification based on the LDA model.

[0118] S64. Explore feature engineering, including feature extraction and selection of documents, combining document content, topic distribution and other features to build a richer and more effective feature representation. At the same time, consider fusing the output results of different classification algorithms to further improve the overall classification performance.

[0119] S65. Evaluate and verify the classification results, including calculating the classification accuracy, recall rate, F1 value and other indicators, as well as visual analysis of the classification results. By comparing the classification effects of different algorithms and models, determine the best classification strategy, and further optimize the model parameters and algorithm selection;

[0120] S66. Continuously optimize and adjust the archiving structure to provide a more orderly and easy-to-retrieve archival storage structure, introduce a real-time update mechanism, and dynamically adjust the archiving structure as new documents are added and classification results change to maintain its consistency and effectiveness with the actual situation. After using the trained LDA model to automatically classify archives, the classification performance of the model is further optimized through manual verification and feedback mechanisms to enhance its robustness and accuracy. In addition, the introduction of advanced machine learning algorithms such as support vector machines (SVM) and neural networks can perform more detailed and multi-level classifications, improving classification accuracy and effectiveness. The overall performance of the classification system is further optimized by integrating feature engineering and algorithm output results. Ultimately, these measures not only improve the orderliness and retrieval of the archiving structure, but also adapt to the needs of real-time updating of archival information by dynamically adjusting the archiving structure, ensuring the high efficiency and practicality of the archive management system.

[0121] S7. Evaluate the performance of the LDA model through perplexity and topic consistency indicators to verify the model effect and ensure that the model can effectively predict the topic distribution of new documents. In addition to using perplexity and topic consistency indicators, introduce other evaluation indicators (such as precision, recall, F1 value) to conduct a comprehensive evaluation of model performance. At the same time, establish a visual display system for model performance evaluation to intuitively display the evaluation results, so as to quickly identify the advantages and disadvantages of the model.

[0122] The step S7 can be refined as follows:

[0123] S71. Select indicators for evaluating the performance of the LDA model, such as perplexity, topic consistency, precision, recall, F1 value, etc., and prepare evaluation data, including training sets, test sets, and topic distribution data.

[0124] S72. Use the selected indicators to evaluate the performance of the LDA model, calculate the perplexity, topic consistency, precision, recall, and F1 value, analyze the performance of the model on the training and test sets, and comprehensively evaluate the model performance.

[0125] S73. Analyze the evaluation results, identify the strengths and weaknesses of the model, find room for improvement by comparing the performance of different indicators, dig deep into the reasons behind the performance, and provide direction for model optimization.

[0126] S74. Adjust and optimize model parameters based on evaluation feedback, and improve model performance and generalization ability by adjusting hyperparameters or introducing technical means (such as regularization, feature selection, etc.).

[0127] S75. Establish a visual display system for model performance evaluation, use charts to intuitively display evaluation indicators and performance, and provide an interactive interface for in-depth analysis of model performance.

[0128] S76. Establish a continuous monitoring and feedback mechanism for model performance, regularly track model performance, adjust and optimize the model according to actual application performance, maintain the best state, and collect user feedback for continuous improvement. Through these steps, comprehensive evaluation, visualization, continuous optimization and monitoring of model performance will be carried out to support the continuous improvement and stable operation of the model in actual applications.

[0129] S8. Display the classification results of the LDA model, including topic keyword distribution and cluster visualization. Use visualization tools to create interactive dashboards to display advanced analysis views such as topic evolution and keyword co-occurrence networks. Customize the display results for different user groups to improve the accessibility and ease of use of information.

[0130] The step S8 can be refined as follows:

[0131] S81. Use the LDA model to extract topic keywords and visualize each topic through word cloud or bar chart, so that users can quickly grasp the characteristics of the topic.

[0132] S82. Use clustering algorithms such as K-means to visualize document clusters in two-dimensional or three-dimensional space, identify different topics with colors or shapes, and show topic evolution and keyword co-occurrence through timelines or dynamic charts to reveal document structure and trends.

[0133] S83. Create interactive dashboards that allow users to customize the display results according to their needs, providing simple interfaces or in-depth thematic analysis functions to meet the needs of different user groups.

[0134] S84. Advanced analysis views are embedded in the dashboard, such as topic evolution diagrams and keyword co-occurrence diagrams, which support users to conduct in-depth analysis of specific topics or keywords and discover the patterns behind the data.

[0135] S85. While displaying the classification results, provide result interpretation and intelligent recommendations, simplify the topic or cluster description, recommend related topics or documents based on user behavior, and improve user experience.

[0136] S86. Establish a user feedback mechanism, optimize the visualization system based on feedback, maintain consistency with user needs, and improve user understanding and satisfaction through advanced analytical views and customized displays.

[0137] Finally, it should be noted that the above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A LDA topic model identification method, characterized in that: The following steps are involved: S1. Use Python library to collect high-frequency search keywords of archives and pre-process the collected data, including removing useless punctuation and special symbols, extracting only Chinese characters, and performing text segmentation. At the same time, crawler technology is used to go deep into the Internet or specific databases to collect richer and more comprehensive archive data. Natural language processing (NLP) technology is introduced to conduct deeper semantic understanding and information extraction. Part-of-speech tagging is used to filter non-key words to improve the quality of data and the accuracy of analysis. Data deduplication technology is introduced to ensure the uniqueness and effectiveness of analysis. S2. Use Jieba word segmentation to optimize performance, introduce domain-specific vocabulary libraries and custom dictionaries, use Jieba's Chinese word segmentation library, combine context to segment text, and improve the accuracy and professionalism of word segmentation. At the same time, develop a real-time update mechanism to dynamically adjust the word segmentation strategy as new words emerge and the frequency of use changes. In addition, explore the combined use of multiple word segmentation algorithms, compare the advantages and disadvantages of each algorithm, and select the word segmentation scheme that best suits the current text collection; S3. Based on the Term Frequency-Inverse Document Frequency (TF-IDF) model, the text is converted into a bag-of-words model, and the TF-IDF matrix is ​​used to create a vocabulary. The accuracy of topic template mining with characteristic words is improved through the two indicators of term frequency and inverse document frequency. In the process of building the bag-of-words model, the language model and deep learning technology are introduced to further capture and understand the deep semantic relationship in the text and enhance the model's expression ability. In addition, the TF-IDF model is optimized and the influence of different parameter settings on the model effect is explored to achieve the best performance. S4. Use the LDA (Latent Dirichlet Allocation) topic model to extract topics, and determine the optimal number of topics by calculating the PV (perplexity-variance) indicator, so as to determine the most appropriate number of topics that can reflect the topic structure of the document collection. When using the LDA topic model to extract topics, combine other variants of the topic model (such as the dynamic topic model DTM, structured topic model STM) to explore the evolution of topics over time or the potential structural relationship between topics. When determining the optimal number of topics, introduce more mathematical indicators and statistical tests, and make a comprehensive judgment through multiple indicators to determine the most reasonable number of topics, so as to improve the accuracy and reliability of topic extraction. S5. Based on the determined optimal number of topics, an LDA model is constructed and trained. This process involves learning topic distribution from documents and word distribution corresponding to topics. The training of the model aims to capture the potential topic structure in text data. In the LDA model training process, cross-validation and grid search mechanisms are introduced to optimize the model's hyperparameter selection. At the same time, parallel computing technology is explored to speed up the model training process. In addition, external knowledge bases or pre-trained models are integrated to deepen the model's understanding and adaptability to specific fields, so as to improve the training effect. S6. Automatically classify and file the archives by theme based on the training results of the LDA model. After the trained LDA model completes automatic classification, further introduce an artificial verification and feedback mechanism for the classified results to continuously optimize the classification model. At the same time, explore the use of machine learning algorithms (such as support vector machine SVM, neural network) for multi-level and refined classification to provide a more orderly and easily retrievable filing structure. S7. Evaluate the performance of the LDA model through perplexity and topic coherence metrics to verify the model effect and ensure that the model can effectively predict the topic distribution of new documents. In addition to using perplexity and topic coherence metrics, introduce other evaluation metrics (such as precision, recall, F1 value) for comprehensive evaluation of the model performance. At the same time, establish a visual display system for model performance evaluation to intuitively display the evaluation results and facilitate quick identification of the advantages and disadvantages of the model. S8. Display the classification results of the LDA model, including the distribution of topic keywords and clustering visualization. By using visualization tools to create an interactive dashboard, display advanced analysis views such as topic evolution and keyword co-occurrence network. Customize the display results for different user groups to improve the accessibility and usability of information.

2. The LDA topic model identification method according to claim 1, characterized in that: The step S1 can be further divided into: S11. Use the Requests and BeautifulSoup libraries in Python to crawl data from the network or a specific database. This step includes writing accurate web requests, parsing the returned HTML or JSON data to obtain the required information. The key to this step is to locate and correctly extract keywords or phrases in the documents, archives or web pages. S12. After obtaining the data, perform preliminary data cleaning and preprocessing, including removing non-content parts such as HTML tags, JavaScript code, CSS styles, etc., then剔除 some useless punctuation marks and special characters, and extract Chinese or key words for analysis. S13. Use the Jieba word segmentation library to perform Chinese word segmentation on the cleaned text. Jieba word segmentation can automatically adjust the dictionary according to word frequency and accurately separate the effective words in the article. S14. By comparing with a custom stop word dictionary, filter out words that have no practical meaning in the analysis, such as "de", "shi", "zai", etc., so as to reduce the noise in subsequent processing and improve the attention and accuracy of data analysis. S15. Use natural language processing technology to further expand and understand the semantics of the word segmentation results, including part-of-speech tagging and semantic understanding, and can perform customized processing on keywords in specific fields to improve the professionalism and accuracy of word segmentation. S16. After all preprocessing work is completed, perform deduplication on the data to ensure the uniqueness and effectiveness of the analysis data. In addition, it is also necessary to reasonably integrate and store the processed data for subsequent data analysis and mining work.

3. The LDA topic model identification method according to claim 1, characterized in that: The step S2 can be further divided into: S21. In the initial stage of large-scale text data processing, Jieba is first used to perform basic text segmentation operations. Since Jieba supports user-defined dictionaries, corresponding professional vocabulary libraries are prepared in advance according to the specific fields of archives, including government affairs, education, medical care, etc. By importing these professional vocabulary libraries into the Jieba word segmenter, the recognition rate of professional terms can be significantly improved, thereby ensuring the accuracy and professionalism of word segmentation to a certain extent; S22. As time goes by and language develops, new words will continue to appear in the language. Therefore, it is necessary to develop a dynamic update mechanism to regularly capture new words and their usage frequency from the Internet or industry literature and update them to the dictionary of the word segmentation system. An automated vocabulary update mechanism can be implemented through API interfaces or crawler technology, so that the word segmentation strategy can respond to actual changes in language use in a timely manner and maintain word segmentation accuracy and timeliness. S23. In order to further improve the accuracy of word segmentation, the meaning of words in specific contexts must be considered. For example, "apple" may mean both fruit and a technology company. By analyzing the contextual information before and after the word, the specific meaning of the word can be more accurately determined and a more accurate word segmentation decision can be made. NLP technologies such as part-of-speech tagging and dependency syntax analysis can help achieve this. S24. A single word segmentation algorithm is difficult to handle all types of text data. Therefore, we explored the combination of Jieba with other word segmentation tools or algorithms (such as HMM, CRF and other models) to achieve better word segmentation results. Through the preset test set, we compared the performance of different word segmentation algorithms or algorithm combinations in terms of accuracy and efficiency, and selected the optimal word segmentation solution to apply to the project. S25. After word segmentation, it is an important step to improve the quality of word segmentation to reasonably filter stop words (such as auxiliary words, conjunctions, etc.). Develop an automated stop word processing mechanism, dynamically adjust and optimize the stop word library in combination with the characteristics of the specific field of archives, and use the generated stop word library to filter the word segmentation results, excluding those words that have no actual semantic contribution, so as to improve the accuracy of subsequent text processing and analysis; S26. In order to cope with the processing needs of large-scale text data, it is also necessary to optimize the performance of word segmentation operations from a technical level, and use multi-threading, parallel processing and other technical means to improve the processing speed of word segmentation. At the same time, the data structure generated after word segmentation is optimized to ensure efficient access and processing of data in subsequent processing.

4. The LDA topic model identification method according to claim 1, characterized in that: The step S3 can be subdivided into: S31. In the initial stage, the TF-IDF model is used to convert the text into a bag-of-words model. By considering the statistics of word frequency and inverse document frequency, the recognition of important words in the document is strengthened. The model is built using TfidfVectorizer from the scikit-learn library, and the model performance is optimized by adjusting parameters (such as max_df and min_df, etc.), and common and rare words in the text are eliminated to highlight the distinguishing keywords. S32. Combining word embedding and deep learning techniques (such as Word2Vec or BERT) into the traditional TF-IDF model can supplement the semantic information lost by TF-IDF due to ignoring word order and context, and improve the model's ability to capture the deep semantics of the text; S33. Through experiments and verification of different parameter settings, find the optimal parameter combination, and apply techniques such as cross-validation to evaluate the effect of the model under different parameter configurations to optimize the final text processing performance; S34. Customize the stop word list for specific data sets and text content. Considering the particularity of the archives that may contain professional terms and common words, appropriately adjust the stop word list to suppress or emphasize the weight of certain words, thereby improving the overall analysis accuracy and effectiveness; S35. Use the LDA model to automatically identify and extract topics from the document set, determine the optimal number of topics by calculating the model perplexity and variance index (PV index), optimize the model settings through statistical and visualization methods, and verify the stability and accuracy of the model through topic consistency testing; S36. After the model parameters are determined, a final model performance evaluation is conducted to ensure that the model can accurately classify and archive files, and the versatility and practicality of the model are tested through actual application scenarios. Final adjustments and optimizations are made based on feedback. Through this step, it is ensured that the text analysis model exhibits good stability and efficiency not only during the research period but also in actual applications.

5. The LDA topic model identification method according to claim 1, characterized in that: The step S4 can be refined as follows: S41. Based on the LDA model, consider introducing the dynamic topic model (DTM) and the structured topic model (STM) to capture the dynamic characteristics of the document collection evolving over time or explore the potential structural relationship between topics. Preliminary model parameters are set, such as the number of topics and the number of iterations. These parameters will be further optimized in subsequent steps. S42. Preprocess the data according to the requirements of the LDA model, including text cleaning (removing stop words, punctuation marks, etc.), word segmentation, and building word bags. Use advanced text processing techniques, such as TF-IDF conversion, to reduce the impact of high-frequency non-topic words and enhance the recognition effect of the model. S43. Use LDA and its variant models to extract the topic of the document, and conduct preliminary topic interpretation to evaluate the rationality and accuracy of the topic. This step can be performed using graphical tools such as PyLDAVis. S44. Combine the perplexity variance (PV) indicator to adjust the number of topics and find the balance between minimizing perplexity and maximizing topic variance. In addition, consider introducing other mathematical indices such as document consistency test to evaluate the discrimination and consistency of topics. S45. Compare the results under different models and settings, not only limited to PV indicators, but also including indicators such as topic consistency and document uniqueness. Combine the results of multiple indicators and use statistical methods to determine the optimal topic model parameters; S46. In specific application scenarios, such as digitalization of archives management and government information systems, the optimized model is deployed for actual application testing, and the model parameters are further adjusted based on usage feedback. The theoretical design and practical performance of the model are evaluated through the comprehensive performance of actual scenarios.

6. The LDA topic model identification method according to claim 1, characterized in that: The step S5 can be refined as follows: S51. Before starting to build the LDA model, you first need to determine the optimal number of topics, which can be achieved by using the aforementioned PV indicator and other methods. At the same time, prepare the data, including preprocessing steps such as data cleaning, word segmentation, and word bag construction to ensure the quality and integrity of the data; S52. After determining the optimal number of topics, start building the LDA model and set the model's hyperparameters, such as the number of iterations, α and β values, etc. The selection of these parameters will directly affect the performance and effect of the model, which needs to be adjusted and optimized according to the specific situation; S53, introduce cross-validation technology and grid search mechanism, through repeated training of the model and evaluation on the validation set, select the best hyperparameter combination, so as to avoid overfitting and improve the generalization ability of the model, so that it performs better on new data; S54. Consider using parallel computing technology to accelerate the model training process, especially when processing large-scale data, which can significantly shorten the training time, including using multi-core CPU parallel processing or GPU acceleration to improve training efficiency; S55. Integrate external knowledge bases or pre-trained models into LDA model training to improve the model's understanding and adaptability to specific fields, including introducing domain-specific vocabulary libraries, terminology dictionaries, or using pre-trained word vector models, thereby improving the model's expressiveness and training results; S56. After the model training is completed, the model is evaluated and the results are verified, including checking the performance of the model on the training set and the test set, and verifying the rationality of the topic distribution and keyword distribution. The model is comprehensively evaluated through a variety of evaluation indicators and visualization tools to ensure that its performance and effects meet the expected requirements.

7. The LDA topic model identification method according to claim 1, characterized in that: The step S6 can be refined as follows: S61. Based on the trained LDA model, the archives are automatically classified and filed according to themes, and each document is mapped to the most relevant theme, thereby achieving automatic classification. After the classification is completed, preliminary manual verification is performed to ensure the accuracy and completeness of the classification results; S62. Introduce a manual verification and feedback mechanism after classification, feed the manual verification results back to the model to optimize the classification model, and continuously improve and optimize the LDA model by collecting feedback information from manual verification to improve its classification accuracy and robustness; S63. Explore the use of machine learning algorithms for multi-level and refined classification. In addition to the LDA model, consider introducing support vector machine (SVM), neural network and other machine learning algorithms to build more complex and accurate classification models. These algorithms can further improve the accuracy and effect of classification based on the LDA model. S64. Explore feature engineering, including feature extraction and selection of documents, combining document content, topic distribution and other features to build a richer and more effective feature representation. At the same time, consider fusing the output results of different classification algorithms to further improve the overall classification performance. S65. Evaluate and verify the classification results, including calculating the classification accuracy, recall rate, F1 value and other indicators, as well as visual analysis of the classification results. By comparing the classification effects of different algorithms and models, determine the best classification strategy, and further optimize the model parameters and algorithm selection; S66. Continuously optimize and adjust the archiving structure to provide a more orderly and easily searchable archive storage structure, introduce a real-time update mechanism, and dynamically adjust the archiving structure as new documents are added and classification results change to maintain its consistency and effectiveness with the actual situation.

8. The LDA topic model identification method according to claim 1, characterized in that: The step S7 can be refined as follows: S71. Before evaluating the performance of the LDA model, you first need to select appropriate evaluation indicators. In addition to the perplexity and topic consistency indicators, you can also consider introducing other indicators, such as precision, recall rate, and F1 value, to comprehensively evaluate the performance of the model. At the same time, prepare the data required for the evaluation, including the division of training sets and test sets, as well as the corresponding topic distribution data; S72. Use the selected evaluation indicators to evaluate and verify the performance of the LDA model. By calculating indicators such as perplexity and topic consistency, analyze the performance of the model on the training set and the test set, so as to verify the model's ability to predict the topic distribution of new documents. At the same time, use indicators such as precision, recall rate, and F1 value to comprehensively evaluate the model performance and have a more comprehensive understanding of the model's performance. S73. Analyze the evaluation results to understand the strengths and weaknesses of the model. By comparing the performance of different indicators on the training set and the test set, find out the problems and room for improvement of the model. At the same time, explain the evaluation results, dig deep into the reasons behind the model performance, and provide guidance and suggestions for further optimization. S74. Adjust and optimize the parameters of the model according to the evaluation results. Further improve the performance and generalization ability of the model by adjusting the hyperparameters of the LDA model or introducing other technical means, such as model regularization and feature selection. This step requires combining the actual situation and evaluation feedback to flexibly adjust the model to achieve better results. S75. Establish a visualization display system for model performance evaluation to intuitively display the evaluation results. Through visualization tools, the evaluation indicators, performance on the training set and the test set are displayed in the form of charts to facilitate rapid identification of the advantages and disadvantages of the model. At the same time, an interactive interface is provided to facilitate users to gain an in-depth understanding of the model performance and conduct further analysis and discussion. S76. Establish a continuous monitoring mechanism to regularly track and provide feedback on model performance. By monitoring the performance of the model in actual applications and taking timely adjustment and optimization measures, ensure that the model performance is always in the best state. At the same time, accept user feedback, continuously improve the model, and enhance its applicability and stability.

9. The LDA topic model identification method according to claim 1, characterized in that: The step S8 can be refined as follows: S81. Extract keywords for each topic through the LDA model and determine the characteristic description of each topic. Then, use visualization tools such as word cloud diagrams or keyword bar charts to intuitively display the keyword distribution of each topic, which helps users quickly understand the main content and characteristics of each topic. S82. Cluster documents by topic and visualize them in two-dimensional or three-dimensional space using clustering algorithms (such as K-means). Encode different topics by color or shape to show the relationship and evolution trend between topics. At the same time, combine timelines or dynamic charts to show the changes of topics over time and the co-occurrence of keywords between different topics, thereby revealing the internal structure and development trend of the document collection. S83. Create interactive dashboards that allow users to freely explore document classification results and topic distribution through dragging, zooming, and other operations. Customize the display results according to the needs of different user groups, provide personalized data views and function selections, provide deeper topic analysis and detailed display for some professional users, and provide simple and clear interfaces and intuitive and easy-to-understand chart displays for ordinary users; S84. Embed advanced analytical views in the dashboard, such as topic evolution diagrams and keyword co-occurrence network diagrams, to deeply explore the intrinsic structure and relevance of document collections. Through interactive operations, users can conduct in-depth analysis on specific topics or keywords and discover the patterns and insights hidden behind the data. S85. While displaying the classification results, provide corresponding result interpretation and intelligent recommendation. Through text summarization, key information extraction and other technologies, briefly describe and summarize each topic or cluster to help users quickly understand the document content. At the same time, intelligently recommend related topics or documents based on the user's browsing behavior and interest preferences to improve information accessibility and user experience. S86. Establish a user feedback mechanism to collect users’ opinions and suggestions on the display results. Based on user feedback, continuously optimize the visualization display system to improve its usability and user satisfaction. Regularly update system functions and interface design to keep pace with user needs and achieve effective communication and understanding of information.

Images