File batch reading method and system based on text mining

Abstract

The invention discloses a file batch reading method and system based on text mining, in a text preprocessing stage, a large number of unstructured data sets can be efficiently processed, and original data is converted into a structured format by executing operations such as data cleaning, noise filtering, text word segmentation and feature extraction; in the analysis modeling stage, according to the specific characteristics of problems, a suitable machine learning or deep learning algorithm can be automatically screened and applied, and structured data is converted into text analysis insight, so that deep understanding and efficient utilization of text content are realized; in the evaluation and interpretation stage, quantitative evaluation is carried out on the performance of the analysis model by selecting proper evaluation indexes, and detailed optimization and adjustment are carried out on the model according to an evaluation feedback result to ensure that abstract text analysis insight is converted into clear text knowledge; in the result display stage, the analyzed and mined result is presented in a user-friendly form, and visual display and convenient operation of text knowledge are supported.

Description

Technical Field

[0001] The present invention belongs to the field of natural language processing, and in particular relates to a method and system for batch reading of files based on text mining. Background Art

[0002] With the rapid development of information technology, the amount of documents that need to be processed on a daily basis has increased exponentially, and the diversity and complexity of document content have increased significantly, posing a major challenge to work efficiency and learning effectiveness. Traditional manual document review methods are inefficient and difficult to meet the current stringent requirements for information processing speed and accuracy. Therefore, designing an automated document processing method to quickly extract key information from a large number of documents has become an urgent need for researchers in various fields. This method should be able to optimize the information retrieval process and improve the efficiency and quality of data processing through an algorithm-driven approach, thereby effectively coping with the current situation of information overload.

[0003] In the field of text mining, the type of problem to be solved is a key factor in deciding which machine learning or deep learning algorithm to use. Diverse document processing needs have given rise to a variety of problem categories, such as: classification problems that map text data to predefined category labels, clustering problems that group text based on similarity in the absence of preset categories, regression problems that predict continuous or real values ​​in text data, sentiment analysis problems that identify and extract sentiment in text, text generation problems that automatically generate coherent and meaningful text content, semantic similarity assessment problems that quantify the degree of semantic proximity between two text fragments, and topic modeling problems that discover potential topic distributions from text collections.

[0004] In response to the above requirements, the effectiveness and accuracy of text mining tasks are ensured by carefully selecting and applying appropriate analysis models. The choice of model should be based on the specific characteristics of the problem, the characteristics of the data set, the expected performance indicators, and the acceptable computational complexity. In actual operation, model tuning and verification may also be required to adapt to specific business scenarios and performance requirements. In current text mining practices, problem definition usually relies on manual settings, and intelligent systems that automatically identify problem features and select appropriate models are relatively rare. Summary of the invention

[0005] In order to solve the shortcomings of the prior art and achieve the purpose of improving the efficiency and accuracy of text information processing, the present invention adopts the following technical solutions:

[0006] The method for batch reading files based on text mining includes the following steps:

[0007] Step S1, text preprocessing module: batch acquisition of unstructured text data and conversion into structured data;

[0008] Step S2, analysis and modeling; determining a text analysis method by identifying the acquired question category; extracting features from the structured data, calculating the importance of each keyword and constructing a TF-IDF value matrix of the keyword, selecting a portion of the keyword matrix with a high TF-IDF value as an analysis object, and constructing a co-occurrence network of keyword pairs with high semantic relevance using an association analysis algorithm;

[0009] Step S3, evaluation and interpretation; identify the categories of problems obtained, and use the social network analysis (SNA) method to construct a cooperation network model among government departments; the nodes represent the publishers of each document, and the size of the node reflects the frequency of publication of the unit, that is, the more the number of publications, the larger the node size, thereby revealing the activity of the department in policy making; the edges between nodes represent the cooperation relationship between units, and the width of the edge is positively correlated with the closeness of cooperation, which is usually determined based on quantitative indicators of cooperation frequency or cooperation intensity, thereby revealing the cooperation relationship and network structure among different government departments. Through social network analysis, key actors, core departments, marginal participants in the policy network, as well as cluster structures and potential cooperation models in the network can be identified, providing a quantitative analysis perspective for understanding the interaction and collaborative work among government departments;

[0010] Therefore, the calculated keyword nodes with high importance are interpreted as hot spots, and the research focus of existing policies and future shipping industry policy trends are summarized and predicted; the units with high cooperation frequency calculated are interpreted as the main units issuing shipping policies, and their influence and authority on the shipping policy field in the process of shipping policy formulation, implementation or research are analyzed;

[0011] Step S4, result display: graphically display the co-occurrence network and graphically display the cooperation network.

[0012] Furthermore, in step S1, the raw data is converted into a structured format through preprocessing, and the preprocessing includes: data cleaning, noise filtering, text segmentation and feature extraction to remove irrelevant characters and eliminate noise. Since Chinese text has no obvious word boundaries, Chinese word segmentation is required to decompose the continuous text sequence into discrete vocabulary units and remove stop words in the text;

[0013] Furthermore, in step S2, the expression of TF value is as follows:

[0014]

[0015] Among them, TF(t, d) represents the original word frequency of word t in document d, TF(d) represents the word frequency of all words in document d, and max(TF(d)) represents the maximum word frequency value in document d;

[0016] When calculating the inverse document frequency, in order to avoid the situation where the denominator is zero, a constant k is usually added to the denominator. The calculation formula of IDF is:

[0017]

[0018] Multiply the term frequency and the inverse document frequency to get the importance score of a term in a specified document, that is, the TF-IDF value:

[0019] TFIDF(t,d,D)=TF(t,d)×IDF(t,D).

[0020] Furthermore, in step S2, a co-occurrence matrix of keywords is constructed using an association analysis algorithm. The co-occurrence matrix is ​​a two-dimensional data structure, and its row index and column index correspond to different keyword sets respectively. Each element C(i, j) in the matrix represents the frequency or co-occurrence intensity of keyword i and keyword j in the text. The co-occurrence intensity of keyword pairs with high semantic relevance can be enhanced based on the number of co-occurrences or through a certain weighting scheme (such as PMI, point mutual information, etc.) through quantitative measurement. Through threshold setting and network simplification strategies, a co-occurrence network is constructed and refined into keyword pairs containing only high co-occurrence intensity, thereby revealing the implicit semantic associations and concept groups in the text data.

[0021] Furthermore, in the co-occurrence network analysis, a network graph is constructed with the keywords in the text as nodes. The size of the node is positively correlated with the co-occurrence intensity of the keywords, that is, the higher the co-occurrence frequency or the co-occurrence significance index obtained by weight calculation (such as TF-IDF value), the larger the node size, which reflects the importance of the keyword in the corpus. The edges in the network represent the co-occurrence relationship between keywords, and the width of the edge is proportional to the co-occurrence intensity, which intuitively represents the semantic association strength between keyword pairs.

[0022] Furthermore, in the step S3, in the evaluation interpretation, the centrality index in the network is quantified through the constructed social network analysis to identify and evaluate the key individuals or nodes in the network; the degree centrality is used to measure the number of other nodes directly connected to a node, reflecting the extensiveness of the connection of the node in the network; the closeness centrality is used to evaluate the average distance from the node to all other nodes in the network, revealing the geographical or information transmission centrality of the node in the network; the betweenness centrality is used to quantify the bridge role of the node in the network, that is, the intermediary frequency of the node in connecting different parts of the network; the calculation results of these centrality indicators, combined with network visualization technology, can provide a powerful analysis tool for understanding and optimizing the cooperative relationship between organizations.

[0023] Furthermore, degree centrality is used as the basis for node size in the network graph. Degree centrality refers to the number of other nodes connected to a node in the network, that is, the number of neighbor nodes of the node. The higher the degree of a node, the greater its importance in the network. The expression of degree centrality is as follows:

[0024]

[0025] Among them, C D (v) is the degree centrality of node v, deg(v) is the degree of node v, and N is the total number of nodes in the network.

[0026] A system for batch reading of documents based on text mining, including a text preprocessing module, an analysis and modeling module, an evaluation and interpretation module, and a result display module;

[0027] The text preprocessing module is used to collect text data and construct a structured text data set;

[0028] The analysis and modeling module obtains key semantic information to determine the problem domain according to the nature of the problem, automatically selects the corresponding prediction model and the corresponding algorithm, and converts the structured data into text analysis insight information to better understand and use the text data. It can realize the conversion of structured data into text analysis insight information to achieve in-depth analysis and effective application of text content; in the analysis and modeling module, the automatic selection mechanism includes classification and clustering, entity recognition, association analysis, trend prediction, sentiment analysis, and topic modeling;

[0029] The evaluation and interpretation module identifies the acquired question categories, selects corresponding evaluation indicators to evaluate the model effect, and further optimizes and adjusts the deployment and application of the model based on the evaluation results, thereby converting the obscure text analysis insight information into clear text knowledge;

[0030] In the evaluation and explanation module, the evaluation and explanation include model performance test verification, error analysis, application evaluation and result explanation. To ensure the accuracy of prediction, identify and correct model defects, evaluate the practical application value of the model, and clearly convey the analysis insights; select appropriate evaluation indicators according to the task type, such as accuracy, precision, recall, F1 score and confusion matrix of text classification; sentiment analysis is similar, with ROC curve and AUC value attached; named entity recognition focuses on precision, recall and F1 score, as well as entity-level performance; text clustering evaluates clustering quality through silhouette coefficient, Davies-Bouldin index, Calinski-Harabasz index, mutual information, Jaccard index, Rand index and V-Measure. The quality of text summarization is measured by ROUGE score and BLEU score, while topic modeling is evaluated by perplexity and consistency score. Model optimization and adjustment are based on the test set evaluation results, involving strategies such as model structure adjustment, data set expansion and data enhancement. Model deployment involves applying the trained model to the actual environment and monitoring updates to adapt to data changes. Result interpretation is to transform technical output into easy-to-understand information. Application evaluation and optimization: evaluate performance through indicators such as accuracy and recall, and adjust the model based on the results to improve the effect;

[0031] The result display module presents the results of the analysis and mining process to the end user in an easy-to-understand and easy-to-operate manner to achieve a visual display of textual knowledge.

[0032] In the result display module, key evaluation indicators are presented intuitively through visualization technology, including word cloud diagrams of word frequency, sentiment diagrams of sentiment analysis, trend diagrams of time series, and data clustering diagrams; performance indicators such as accuracy, precision, recall, F1 score, etc. are used to quantitatively evaluate the effectiveness of the model, and the result report is used to elaborate on the analysis results and their use value to ensure that users can understand and effectively use them; the analysis results are presented to users in an intuitive and easy-to-understand form, mainly including displaying keyword frequency and importance through word cloud diagrams, revealing the sentiment tendency in the text through sentiment distribution diagrams, trend analysis diagrams showing data changes over time, and clustering result diagrams showing data grouping. Key indicators such as accuracy, precision, recall, F1 score, etc. provide quantitative model performance evaluation, and the result report needs to fully explain the analysis findings and value so that users can understand and apply them.

[0033] Furthermore, in the text preprocessing module, the acquired text data is aggregated and constructed into a unified data set, data cleaning is performed, and whitespace characters are normalized to improve the accuracy and efficiency of subsequent analysis.

[0034] Furthermore, in the text preprocessing module, the jieba word segmentation library is used to perform lexical analysis on Chinese text, and the continuous text is divided into discrete vocabulary units through word segmentation processing, and a custom stop word set is applied to filter out functional words, that is, those words that appear frequently but have low information entropy, thereby reducing the data dimension and enhancing the targeted analysis.

[0035] The advantages and beneficial effects of the present invention are:

[0036] The document batch reading method and system based on text mining proposed in the present invention can quickly analyze a large number of documents, extract key information, and output customized reports according to different needs of users through automated processing and intelligent algorithms, thereby significantly improving the efficiency and accuracy of information processing. The present invention can quickly extract valuable information from a large number of text files according to the specific needs of users, significantly reducing the need for manual reading, and effectively improving the efficiency bottleneck of traditional document review methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is an architecture diagram of the system in an embodiment of the present invention.

[0038] Figure 2 4 is a flowchart of text preprocessing in an embodiment of the present invention.

[0039] Figure 3 It is a flow chart of analysis and modeling in an embodiment of the present invention.

[0040] Figure 4 Detailed description of the invention

[0041] Figure 5 It is a flow chart showing the results in an embodiment of the present invention.

[0042] Figure 6 It is a high-frequency word co-occurrence network diagram in an embodiment of the present invention.

[0043] Figure 7 It is a cooperation network diagram in an embodiment of the present invention.

[0044] Figure 8 It is a high-frequency keyword cloud diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0046] like Figure 1As shown in the figure, the batch reading system for documents based on text mining includes a text preprocessing module, an analysis and modeling module, an evaluation and interpretation module, and a result display module. The text preprocessing module is used to structure text information, the analysis and modeling module is used to analyze structured data, the evaluation and interpretation module is used to evaluate and deploy the model effect, and the result display module is used to visualize text knowledge.

[0047] Text preprocessing module:. Data collection, aggregating the text data submitted by users to build a unified data set. Implement data cleaning, using regular expression algorithms to remove HTML / XML tags, eliminate insignificant special symbols, non-critical numbers, and perform normalization of whitespace characters to improve the accuracy and efficiency of subsequent analysis. For Chinese text, use the jieba word segmentation library for lexical analysis, splitting continuous text into discrete lexical units through word segmentation, and applying a custom stop word set to filter out function words, that is, those words with high frequency but low information entropy, such as "de", "he", "shi", etc., thereby reducing the data dimension and enhancing the pertinence of analysis. Combine TF-IDF and Word2Vec for feature vectorization. TF-IDF quantifies the importance of words based on word frequency and inverse document frequency; Word2Vec generates word vectors through neural networks to reveal semantic associations between words. Multidimensional feature representation improves the accuracy of key concept recognition and deepens semantic understanding.

[0048] Analysis and modeling module: The system captures user requirements, obtains key semantic information to determine the problem domain, and determines the nature of problems such as supervised learning and unsupervised learning, classification regression, or clustering. Screen suitable algorithms such as logistic regression, support vector machine, neural network, K-means, hierarchical clustering, etc. Perform feature engineering, feature construction, screening, and transformation to improve the prediction accuracy of the model. The model trains to learn the data pattern and ensures generalization ability through leave-one-out cross-validation. In this way, each sample is independently used to verify the model, which helps to reduce bias in the evaluation process and provides a robust estimate of the model performance.

[0049] Evaluation and interpretation module: Select appropriate evaluation indicators according to the task type, such as accuracy, precision, recall, F1 score and confusion matrix for text classification; sentiment analysis is similar, with additional ROC curve and AUC value; named entity recognition focuses on precision, recall and F1 score, as well as entity-level performance; text clustering evaluates clustering quality through silhouette coefficient, Davies-Bouldin index, Calinski-Harabasz index, mutual information, Jaccard index, Rand index and V-Measure. The quality of text summarization is measured by ROUGE score and BLEU score, while topic modeling is evaluated by perplexity and consistency score. Model optimization and adjustment is based on the test set evaluation results, involving strategies such as model structure adjustment, data set expansion and data enhancement. Model deployment involves applying the trained model to the actual environment and monitoring updates to adapt to data changes. Result interpretation is to convert technical output into easy-to-understand information. Application evaluation and optimization evaluates performance through indicators such as accuracy and recall, and adjusts the model based on the results to improve the effect.

[0050] Results display module: presents the analysis results to users in an intuitive and easy-to-understand form, mainly including displaying keyword frequency and importance through word cloud diagrams, revealing the emotional tendency in the text through sentiment distribution diagrams, trend analysis diagrams showing data changes over time, and clustering result diagrams showing data grouping. Key indicators such as accuracy, precision, recall, F1 score, etc. provide quantitative model performance evaluation, and the result report needs to fully explain the analysis findings and value so that users can understand and apply them.

[0051] The method for batch reading files based on text mining includes the following steps:

[0052] Step S1, constructing a text preprocessing module: converting the collected unstructured data into structured data in batches through steps such as cleaning, noise reduction, word segmentation, and feature engineering for further data analysis.

[0053] Specifically, the raw data is converted into a structured format through preprocessing, including: data cleaning, noise filtering, text segmentation, and feature extraction.

[0054] Step S2: Build an analysis modeling module: Automatically select a suitable machine learning or deep learning model based on the nature of the problem to transform structured data into text analysis insights to better understand and utilize text data.

[0055] Specifically, it automatically identifies and selects appropriate machine learning or deep learning technologies, including classification and clustering, entity recognition, association analysis, trend prediction, sentiment analysis, topic modeling, etc. It can transform structured data into insights for text analysis, so as to achieve in-depth analysis and effective application of text content.

[0056] Step S3, construct an evaluation and interpretation module: select appropriate evaluation indicators to evaluate the model effect according to different problems, and further optimize and adjust the deployment and application of the model based on the evaluation results, so as to transform the obscure text analysis insights into clear text knowledge.

[0057] Specifically, evaluation and interpretation include testing and verification of model performance, error analysis, application evaluation, and result interpretation. To ensure prediction accuracy, identify and correct model defects, evaluate the practical application value of the model, and clearly communicate analytical insights.

[0058] Step S4, constructing a result display module: presenting the results of the analysis and mining process to the end user in an easy-to-understand and easy-to-operate manner to achieve a visual display of textual knowledge.

[0059] Specifically, the results display uses visualization technology to intuitively present key analysis indicators, including word cloud diagrams for word frequency, sentiment diagrams for sentiment analysis, trend diagrams for time series, and data clustering diagrams. Performance indicators such as accuracy, precision, recall, and F1 scores quantitatively evaluate the effectiveness of the model, and the result report details the analysis results and their use value to ensure that users can understand and effectively use them.

[0060] In this example, the data collection and preprocessing phase selects the Peking University Law Database as the collection point for policy data, and uses the keyword "shipping" to perform literature retrieval. After manual screening and removal of irrelevant documents, a total of 223 central government policy texts related to the topic were collected from 1986 to 2023. The collected documents were then downloaded and uniformly saved in text format (.txt) for subsequent text analysis and data processing.

[0061] In the preprocessing stage of policy text analysis, such as Figure 2 As shown, first, the collected policy text file is imported into the text preprocessing module of step S1, which will perform the removal of irrelevant characters, including punctuation marks, numbers and other non-semantic characters, to eliminate noise that may interfere with the subsequent text analysis algorithm. Since Chinese text does not have obvious word boundaries, Chinese word segmentation is required to decompose the continuous text sequence into discrete vocabulary units. Subsequently, the stop words in the text are removed. These are usually functional words in the language, such as prepositions, conjunctions, auxiliary words, etc., which do not carry important semantic information in text analysis.

[0062] Identify and confirm the categories of the research issues. First, identify the keywords in the task content as "policy hot spots" and "departmental cooperation". Through keyword search and combined with the existing task library, it is preliminarily determined that the analysis methods that may be involved are association analysis, cluster analysis, and topic modeling. Verify with the user and confirm that the final analysis task is association analysis and cluster analysis.

[0063] After cleaning and selecting, feature extraction is performed on the policy text and the importance of each keyword is calculated using the TF-IDF algorithm. The TF-IDF value matrix of the keyword is constructed, where the expression of the TF value is as follows:

[0064]

[0065] Among them, TF(t, d) is the original word frequency of word (t) in document (d), (TF(d)) is the word frequency of all words in document (d), and max(TF(d)) is the maximum word frequency value in document (d).

[0066] When calculating the inverse document frequency, in order to avoid the situation where the denominator is zero, a constant (k) is usually added to the denominator, and this constant is usually 1. In this way, the calculation formula of IDF becomes:

[0067]

[0068] Multiply the term frequency and the inverse document frequency to get the importance score of a term in a specified document, that is, the TF-IDF value:

[0069] TFIDF(t,d,D)=TF(t,d)×IDF(t,D)

[0070] According to the research purpose, some keyword matrices with high TF-IDF values ​​were selected as the analysis objects. At the same time, the publishing units of each policy text were counted, and the co-occurrence matrix of government department cooperation was constructed to analyze the cooperation relationship between departments.

[0071] Import the TF-IDF value matrix of keywords into the analysis and modeling module of step S2, such as Figure 3As shown in the figure, this module uses the association analysis algorithm to construct a keyword co-occurrence matrix (Co-occurrenceMatrix) to form a co-occurrence network. The co-occurrence matrix is ​​a two-dimensional data structure whose row index and column index correspond to different sets of keywords. Each element (C(i, j)) in the matrix represents the frequency or co-occurrence intensity of keyword (i) and keyword (j) in the text. This quantitative measure can enhance the co-occurrence intensity of keyword pairs with high semantic relevance based on the number of co-occurrences or through some weighting scheme (such as PMI, point mutual information, etc.). Through threshold setting and network simplification strategies, the co-occurrence network is further refined to contain only keyword pairs with high co-occurrence intensity, thereby revealing the implicit semantic associations and concept groups in the text data.

[0072] In the co-occurrence network analysis, the nodes in the constructed network graph represent the keywords in the text, and the size of the node is positively correlated with the co-occurrence strength of the keywords, that is, the higher the co-occurrence frequency or the co-occurrence significance index (such as TF-IDF value) obtained by weight calculation, the larger the node size, which reflects the importance of the keyword in the corpus. The edges in the network represent the co-occurrence relationship between keywords, and the width of the edge is proportional to the co-occurrence strength, which intuitively represents the semantic association strength between keyword pairs.

[0073] Using the Social Network Analysis (SNA) method, a cooperative network model between government departments can be constructed. In the cooperative network diagram, the nodes represent the issuing units of each policy document, and the size of the node reflects the frequency of publication of the unit, that is, the more the number of publications, the larger the node size, thus revealing the activity of the department in policy making. The edges between nodes represent the cooperative relationship between departments, and the width of the edge is positively correlated with the closeness of cooperation. It is usually determined based on quantitative indicators of cooperation frequency or cooperation intensity, so as to reveal the cooperative relationship and network structure between different government departments. Through social network analysis, key actors, core departments, marginal participants in the policy network, as well as cluster structures and potential cooperation models in the network can be identified, providing a quantitative analytical perspective for understanding the interaction and collaborative work between government departments.

[0074] like Figure 4As shown in the figure, in the evaluation and interpretation module of step S3, social network analysis (SNA) is implemented to quantify the centrality index in the network, aiming to identify and evaluate key individuals or nodes in the network. Specifically, degree centrality measures the number of other nodes directly connected to a node, reflecting the extensiveness of the node's connections in the network. Closeness centrality evaluates the average distance from a node to all other nodes in the network, revealing the node's central position in the network in terms of geography or information transmission. Betweenness centrality quantifies the bridge role of a node in the network, that is, the frequency of the node's intermediary role in connecting different parts of the network. The calculation results of these centrality indicators, combined with network visualization technology, can provide a powerful analytical tool for understanding and optimizing cooperative relationships between organizations. Among them, degree centrality is used as the basis for the size of nodes in the network graph. Degree centrality refers to the number of other nodes connected to a node in the network, that is, the number of neighbor nodes of the node. The higher the degree of a node, the greater its importance in the network. The expression of degree centrality is as follows:

[0075]

[0076] Among them, (C D (v)) is the degree centrality of node (v), (deg(v)) is the degree of node (v), and (N) is the total number of nodes in the network.

[0077] The key word nodes with high calculated importance are interpreted as policy hot spots, and the research focus of existing policies and future shipping industry policy trends are summarized and predicted. The departments with high calculated cooperation frequency are regarded as the main departments for shipping policy release, and their influence and authority on the shipping policy field in the process of shipping policy formulation, implementation or research are analyzed.

[0078] like Figure 5 As shown in Figure 2, in the result display module of step S4, high-frequency words are displayed through word cloud diagrams and the co-occurrence network constructed is graphically displayed using the network visualization tool Gephi to realize the knowledge visualization of text data. The association analysis and cluster analysis are deeply interpreted through visualization charts to reveal policy hotspots and key models of inter-departmental cooperation, such as Figures 6 to 8 As shown. Based on the analysis results, targeted policy recommendations are put forward to improve the efficiency of shipping policy formulation and implementation. In addition, the analysis results are applied to key links such as policy evaluation, formulation and communication to enhance the effectiveness and transparency of policies and ensure the scientificity and rationality of the policy decision-making process.

[0079] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A file batch reading method based on text mining, characterized by The steps include: Step S1, text preprocessing module: batch acquisition of unstructured text data and conversion into structured data; Step S2, analysis and modeling; determining a text analysis method by identifying the acquired question category; extracting features from the structured data, calculating the importance of each keyword and constructing a TF-IDF value matrix of the keyword, selecting a portion of the keyword matrix with a high TF-IDF value as an analysis object, and constructing a co-occurrence network of keyword pairs with high semantic relevance using an association analysis algorithm; Step S3, evaluation explanation; identifying the acquired problem categories, using social network analysis methods to construct a cooperation network model; the nodes represent publishers of each document, the size of the node reflects the frequency of publication of the unit, the edges between the nodes represent the cooperation relationship between the units, and the width of the edge is positively correlated with the closeness of the cooperation; Step S4, result display: graphically display the co-occurrence network and graphically display the cooperation network.

2. The method for batch reading files based on text mining according to claim 1 is characterized in that: In step S1, the raw data is converted into a structured format through preprocessing, and the preprocessing includes: data cleaning, noise filtering, text segmentation and feature extraction to remove irrelevant characters, eliminate noise, perform Chinese word segmentation, decompose continuous text sequences into discrete vocabulary units, and remove stop words in the text.

3. The method for batch reading files based on text mining according to claim 1 is characterized in that: In step S2, the expression of TF value is as follows: Among them, TF(t, d) represents the original word frequency of word t in document d, TF(d) represents the word frequency of all words in document d, and max(TF(d)) represents the maximum word frequency value in document d; When calculating the inverse document frequency, add the constant k to the denominator, and the IDF calculation formula is: Multiply the term frequency and the inverse document frequency to get the importance score of a term in a specified document, that is, the TF-IDF value: TFIDF(t,d,D)=TF(t,d)×IDF(t,D).

4. The method for batch reading files based on text mining according to claim 1 is characterized in that: In the step S2, a co-occurrence matrix of keywords is constructed by using an association analysis algorithm. The co-occurrence matrix is ​​a two-dimensional data structure, and its row index and column index correspond to different keyword sets respectively. Each element C(i, j) in the matrix represents the frequency or co-occurrence intensity of keyword i and keyword j in the text. The co-occurrence intensity of keyword pairs with high semantic relevance is enhanced through quantitative measurement. Through threshold setting and network simplification strategy, a co-occurrence network is constructed and refined to contain only keyword pairs with high co-occurrence intensity.

5. The method for batch reading files based on text mining according to claim 4 is characterized in that: In the co-occurrence network analysis, a network graph is constructed with the keywords in the text as nodes. The size of the node is positively correlated with the co-occurrence intensity of the keywords, that is, the higher the co-occurrence frequency or the co-occurrence significance index obtained by weight calculation, the larger the node size. The edges in the network represent the co-occurrence relationship between keywords, and the width of the edge is proportional to the co-occurrence intensity.

6. The method for batch reading files based on text mining according to claim 1, characterized in that: In step S3, in the evaluation interpretation, the centrality index in the network is quantified through the constructed social network analysis to identify and evaluate the key nodes in the network; the degree centrality is used to measure the number of other nodes directly connected to a node, reflecting the extensiveness of the connection of the node in the network; the proximity centrality is used to evaluate the average distance from the node to all other nodes in the network, revealing the node's central position in the network in terms of geography or information transmission; the betweenness centrality is used to quantify the bridge role of the node in the network, that is, the intermediary frequency of the node in connecting different parts of the network.

7. The method for batch reading files based on text mining according to claim 6 is characterized in that: Degree centrality is used as the basis for node size in a network graph. Degree centrality refers to the number of other nodes a node is connected to in the network, that is, the number of neighbor nodes of the node. The higher the degree of a node, the greater its importance in the network. The expression of degree centrality is as follows: Among them, C D (v) is the degree centrality of node v, deg(v) is the degree of node v, and N is the total number of nodes in the network.

8. A system for the method for batch reading of documents based on text mining according to claim 1, comprising a text preprocessing module, an analysis and modeling module, an evaluation and interpretation module and a result display module, characterized in that: The text preprocessing module is used to collect text data and construct a structured text data set; The analysis and modeling module obtains key semantic information to determine the problem domain according to the nature of the problem, automatically selects the corresponding prediction model and corresponding algorithm, and converts structured data into insight information for text analysis; The evaluation and interpretation module identifies the acquired question categories, selects corresponding evaluation indicators to evaluate the model effect, and optimizes and adjusts the deployment and application of the model based on the evaluation results, converting text analysis insight information into text knowledge; The result display module visualizes the text knowledge.

9. The system of claim 8, wherein: In the text preprocessing module, the acquired text data is aggregated to form a unified data set, data cleaning is performed, and normalization of blank characters is performed.

10. The system of claim 8, wherein: In the text preprocessing module, a word segmentation library is used to perform lexical analysis on Chinese text, continuous text is divided into discrete vocabulary units through word segmentation, and a custom stop word set is used to filter out functional words.

Images