Archive association method based on theme association relationship analysis

Abstract

The invention provides an archive association method based on theme association relationship analysis, and relates to the technical field of archive management businesses, and the method comprises the following steps: S1, data preparation and preprocessing: collecting archive data needing to be analyzed, preprocessing the archive data through a natural language processing technology, and then carrying out deep analysis to obtain a processed archive; s2, constructing a model: modeling potential topics in the processed archive set by applying an LDA algorithm, and constructing a layered LDA model; and S3, archive association: calculating the similarity between the processed archives based on the topic distribution output by the hierarchical LDA model. Through comprehensive application of advanced technologies such as natural language processing, topic model and graph theory analysis, a brand-new and efficient archive management and retrieval mode is realized, so that the efficiency and accuracy of archive management are remarkably improved, and richer and deeper possibilities are provided for archive utilization and information mining.

Description

Technical Field

[0001] The present invention relates to the technical field of archive management business, and in particular to an archive association method based on subject association relationship analysis. Background Art

[0002] In today's information age, archive management is essential for knowledge management and information retrieval. Traditional archive management systems mainly rely on manual classification and keyword search, which is inefficient when processing large-scale data and often fails to accurately capture the deep-level associations of archive content. With the development of artificial intelligence and natural language processing technologies, there is a need and possibility to use these technologies for archive management. Especially in government agencies, research institutions, and large enterprises, there is an increasing need for an efficient method that can intelligently identify, analyze, and automatically associate archives.

[0003] With the surge in data volume, traditional methods face many challenges in dealing with complex archive associations, such as information islands, low retrieval efficiency, etc. Therefore, developing a method to associate archives can achieve more efficient and accurate archive management and information retrieval, which is of great significance to improving the efficiency and quality of archive work. Summary of the invention

[0004] The invention aims to provide an archive association method based on thematic association relationship analysis. By comprehensively applying advanced technologies such as natural language processing, topic model, graph theory analysis, etc., a new and efficient archive management and retrieval method is realized, which not only significantly improves the efficiency and accuracy of archive management, but also provides richer and deeper possibilities for archive utilization and information mining.

[0005] In order to solve the above-mentioned problems of information islands and low retrieval efficiency, the present invention provides the following technical solutions: a method for associating archives based on subject association relationship analysis, comprising the following steps:

[0006] Step 1: Data preparation and preprocessing: Collect the archival data that needs to be analyzed, preprocess the archival data through natural language processing technology, and then conduct in-depth analysis to obtain the processed archives.

[0007] Step 2: Building a model: Applying the LDA algorithm to model the potential topics in the processed archive collection, and building a hierarchical LDA model.

[0008] Step 3: File association: Based on the topic distribution output by the hierarchical LDA model, the similarity between the processed files is calculated, thereby achieving automatic association of the processed files.

[0009] Step 4: Constructing an archive association network: Using graph theory principles, the data of the processed archives are stored in Neo4j, and an association network of the processed archives is constructed through Neo4j.

[0010] Step 5. Search application: Users search for specific files based on keywords through the search interface, and the system will intelligently recommend other files related to the searched files.

[0011] Furthermore, in step one, the collected archival data includes archival metadata and original text files.

[0012] Furthermore, in step 1, the archival data preprocessing includes the following steps:

[0013] S101, text cleaning: removing non-text elements in the original text file;

[0014] S102, word segmentation processing: decomposing the original text file content;

[0015] S103, stop word removal: deleting common words in the original text file that have no actual analysis value;

[0016] S104, stem extraction: restore the word to its root form to reduce the impact of word form changes on the analysis;

[0017] S105. Build professional dictionaries: Establish professional dictionaries in specific fields.

[0018] Furthermore, in step S102, the original text file content is decomposed into a set of words and phrases, and Chinese word segmentation is performed on the Chinese content.

[0019] Furthermore, in step 2, the hierarchical LDA model processes conventional vocabulary and professional terms respectively, integrates the domain knowledge provided by professional terms, and reveals the deep-level connections between the different processed archives based on themes, going beyond simple keyword matching to achieve more accurate and comprehensive associations.

[0020] Furthermore, in step 2, the training operation steps of the hierarchical LDA model are as follows:

[0021] S201, parameter setting: determining key parameters of the hierarchical LDA model;

[0022] S202, model training: using the processed archive to train a hierarchical LDA model;

[0023] S203, model evaluation and iteration: evaluating the performance of the hierarchical LDA model to ensure that the model can accurately capture the topics in the text, and making repeated corresponding adjustments based on the evaluation results to achieve optimal performance;

[0024] S204, topic identification: After the training is completed, each topic is associated with a group of words representing the topic content. At the same time, each of the processed files is assigned a topic distribution to show the degree of association.

[0025] Furthermore, in step three, the specific method of automatic association is as follows:

[0026] S301, similarity calculation: using the two topic distributions generated by the hierarchical LDA model of the processed archives, using cosine similarity and Jaccard similarity measurement methods, calculate the similarity between the processed archives;

[0027] S302, file association: according to the similarity results between the processed files, the processed files with high similarity are associated with each other.

[0028] Furthermore, in step S301, the similarity result between the processed archives is: sim=α*sim1+β*sim2, sim represents the archive similarity result, sim1 represents the similarity result calculated based on the topic distribution of the conventional LDA model, sim2 represents the similarity result calculated based on the topic distribution of the professional LDA model, and α and β are the similarity ratios of sim1 and sim2.

[0029] Furthermore, in step 4, the steps of constructing the association network are as follows:

[0030] S401, design graph model: design a model of a graph database;

[0031] S402, importing data: importing the processed archive information into the Neo4j database to create an archive node;

[0032] S403, building associations: building relationships between archives based on the similarities and subject associations between the processed archives obtained through the hierarchical LDA model analysis.

[0033] Furthermore, in step S401, the nodes in the graph in the graph database represent processed archives, the edges represent the association relationship between processed archives, the attributes of each node include archive ID, title and date, and the attributes of each edge are the strength of association.

[0034] The present invention provides an archive association method based on subject association relationship analysis, which has the following beneficial effects: by comprehensively applying advanced technologies such as natural language processing, topic model, graph theory analysis, etc., a new and efficient archive management and retrieval method is realized, which not only significantly improves the efficiency and accuracy of archive management, but also provides richer and more in-depth possibilities for archive utilization and information mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a method for associating archives based on subject association relationship analysis of the present invention;

[0036] Figure 2 The schematic diagram of the file association method based on the subject association relationship analysis of the present invention. DETAILED DESCRIPTION

[0037] See also Figure 1 , 2 The present invention provides a technical solution: a file association method based on subject association relationship analysis, comprising the following steps:

[0038] Step 1: Data preparation and preprocessing: Collect the archival data that needs to be analyzed, preprocess the archival data through natural language processing technology, and then conduct in-depth analysis to obtain the processed archives.

[0039] Step 2: Model building: Apply the LDA algorithm to model the potential topics in the processed archive collection and build a hierarchical LDA model.

[0040] Step 3: File association: Based on the topic distribution output by the hierarchical LDA model, the similarity between the processed files is calculated to achieve automatic association of the processed files.

[0041] Step 4: Construct an archive association network: Using graph theory principles, store the processed archive data in Neo4j, and construct an association network for the processed archives through Neo4j.

[0042] Step 5. Search application: Users search for specific files based on keywords through the search interface, and the system will intelligently recommend other files related to the searched files.

[0043] Specifically, in step one, the collected archival data includes archival metadata and original text files.

[0044] Specifically, in step 1, the archive data preprocessing includes the following steps:

[0045] S101. Text cleaning: remove non-text elements in the original text file;

[0046] S102, word segmentation processing: decomposing the original text file content;

[0047] S103, stop word removal: delete common words in the original text file that have no actual analysis value;

[0048] S104, stem extraction: restore the word to its root form to reduce the impact of word form changes on the analysis;

[0049] S105. Build professional dictionaries: Establish professional dictionaries in specific fields.

[0050] Specifically, in step S102, the original text file content is decomposed into a set of words and phrases, and Chinese word segmentation is performed on the Chinese content.

[0051] Specifically, in step 2, the hierarchical LDA model processes conventional vocabulary and professional terms respectively, integrates the domain knowledge provided by professional terms, and reveals the deep topic-based connections between different processed archives, going beyond simple keyword matching to achieve more accurate and comprehensive associations.

[0052] Specifically, in step 2, the training operation steps of the hierarchical LDA model are as follows:

[0053] S201, parameter setting: determining key parameters of the hierarchical LDA model;

[0054] S202, model training: using the processed archives to train a hierarchical LDA model;

[0055] S203, Model evaluation and iteration: Evaluate the performance of the hierarchical LDA model to ensure that the model can accurately capture the topics in the text, and make repeated adjustments based on the evaluation results to achieve the best performance;

[0056] S204, Topic Identification: After training, each topic is associated with a set of words representing the topic content. At the same time, each processed file is assigned a topic distribution to show the degree of association.

[0057] Specifically, in step three, the specific method of automatic association is as follows:

[0058] S301, similarity calculation: using the two topic distributions generated by the hierarchical LDA model of the processed archives, using the cosine similarity and Jaccard similarity measurement methods, calculate the similarity between the processed archives;

[0059] S302, file association: based on the similarity results between the processed files, the processed files with high similarity are associated with each other.

[0060] Specifically, in step S301, the similarity result between the processed archives is: sim=α*sim1+β*sim2, sim represents the archive similarity result, sim1 represents the similarity result calculated based on the topic distribution of the conventional LDA model, sim2 represents the similarity result calculated based on the topic distribution of the professional LDA model, and α and β are the similarity ratios of sim1 and sim2.

[0061] Specifically, in step 4, the steps for constructing the association network are as follows:

[0062] S401, design graph model: design a model of a graph database;

[0063] S402, importing data: importing the processed archive information into the Neo4j database and creating an archive node;

[0064] S403, building associations: building relationships between archives based on the similarities and subject associations between the processed archives obtained through the hierarchical LDA model analysis.

[0065] Specifically, in step S401, the nodes in the graph database represent processed archives, the edges represent the association relationship between processed archives, the attributes of each node include archive ID, title and date, and the attributes of each edge are the strength of association.

[0066] The method of the embodiment is used for detection and analysis, and compared with the prior art, the following data are obtained:

[0067] Retrieval efficiency Retrieval accuracy Example Higher Higher Prior art Lower Lower

[0068] According to the data in the above table, it can be concluded that when the implementation example is used, a new and efficient archive management and retrieval method is realized by comprehensively applying advanced technologies such as natural language processing, topic models, and graph analysis. This not only significantly improves the efficiency and accuracy of archive management, but also provides richer and more in-depth possibilities for archive utilization and information mining.

[0069] The present invention provides a file association method based on subject association relationship analysis, comprising the following steps:

[0070] Step 1, data preparation and preprocessing: collect the archival data that needs to be analyzed, preprocess the archival data through natural language processing technology, and then conduct in-depth analysis to obtain the processed archives. The collected archival data includes archival metadata and original text files. The archival data preprocessing includes the following steps: S101, text cleaning: remove non-text elements in the original text file; S102, word segmentation: decompose the content of the original text file; S103, stop word removal: delete common words in the original text file that have no actual analysis value; S104, stem extraction: restore words to root form to reduce the impact of word form changes on analysis; S105, build professional dictionaries: establish professional dictionaries in specific fields, decompose the content of the original text file into a collection of words and phrases, and perform Chinese word segmentation on Chinese content.

[0071] The step of data preparation and preprocessing includes processes such as text cleaning, keyword extraction, and semantic analysis. Text cleaning mainly removes irrelevant characters and noisy data to ensure the accuracy of analysis. Keyword extraction uses the TF-IDF algorithm to extract the most representative words from the text. Semantic analysis further deeply understands the text content, identifies archive keywords and context relationships. The preprocessing of archive data mainly targets text file information, and non-text elements are HTML tags, special symbols, etc. Common words without practical analysis value are "of", "and", "is", etc. Stemming is mainly applicable to English. A professional dictionary is constructed in the text preprocessing stage, and this dictionary is used to annotate professional terms in the text to guide the hierarchical LDA model to more accurately capture these important information.

[0072] Step 2: Build a model: Apply the LDA algorithm to model the latent topics in the processed archive collection, and build a hierarchical LDA model. The hierarchical LDA model processes regular words and professional terms respectively, integrates the domain knowledge provided by professional terms, reveals the deep connections based on topics between different processed archives, goes beyond simple keyword matching, and achieves more accurate and comprehensive associations. The training operation steps of the hierarchical LDA model are as follows: S201. Parameter setting: Determine the key parameters of the hierarchical LDA model; S202. Model training: Use the processed archives to train the hierarchical LDA model; S203. Model evaluation and iteration: Evaluate the performance of the hierarchical LDA model to ensure that the model can accurately capture the topics in the text, and make repeated corresponding adjustments according to the evaluation results to achieve the best performance; S204. Topic recognition: After training, each topic is associated with a group of words representing the topic content. At the same time, each processed archive is assigned a topic distribution to show the degree of association.

[0073] LDA is an unsupervised machine learning technology that can discover topics from document collections and automatically classify documents. The LDA model is used to analyze the preprocessed text to identify potential topic distributions. Two LDA models are constructed, namely hierarchical LDA models, in which the conventional LDA model is specifically used to process conventional vocabulary, and the other professional LDA model is specifically used to process professional terms. This structure allows the model to more effectively integrate the domain knowledge provided by professional terms when analyzing document topics. In this way, the system can reveal the deep topic-based connections between different archives, going beyond simple keyword matching to achieve more accurate and comprehensive archive associations. A hierarchical LDA model is constructed, and its architecture is designed to finely manage different vocabulary categories of text content. In this model, one layer is responsible for the analysis of common vocabulary, while the other layer focuses on identifying and interpreting professional terms. The purpose of this two-layer structure is to promote the model to more effectively integrate the depth from domain-specific terms when refining the subject content of the document. Knowledge. In this way, the model can more accurately identify and process the basic language and deep industry knowledge in the document, thereby improving the quality and professionalism of topic analysis. The key parameters of the hierarchical LDA model include the number of topics K (which needs to be pre-set according to actual conditions) and the number of iterations of model training. The conventional hierarchical LDA model uses archival common vocabulary data to identify broad topics based on general vocabulary in the document collection. This layer of analysis focuses on capturing common concepts and discussion points in the document without placing special emphasis on professional terminology. The professional LDA model uses archival professional terminology vocabulary data. In this process, the model learns document-topic distribution and vocabulary-topic distribution. This layer of analysis focuses on the details of domain knowledge and reveals the specific content and differences of each topic through professional terminology. The evaluation indicators are perplexity and topic consistency score to ensure that the model can accurately capture the topics in the text, especially those reinforced by professional terminology. Adjustments include preprocessing steps, model parameters and professional terminology.

[0074] Step 3, archive association: based on the topic distribution output by the hierarchical LDA model, calculate the similarity between the processed archives, so as to realize the automatic association of the processed archives. The specific method of automatic association is as follows: S301, similarity calculation: using the two topic distributions generated by the hierarchical LDA model of the processed archives, the cosine similarity and Jaccard similarity measurement methods are used to calculate the similarity between the processed archives; S302, archive association: according to the similarity results between the processed archives, the processed archives with high similarity are associated with each other. The similarity result between the processed archives is: sim=α*sim1+β*sim2, sim represents the archive similarity result, sim1 represents the similarity result calculated based on the topic distribution of the conventional LDA model, sim2 represents the similarity result calculated based on the topic distribution of the professional LDA model, and α and β are the similarity ratios of sim1 and sim2.

[0075] A similarity threshold can be set. Only when the similarity between archives exceeds this threshold, they are considered to be related. The calculation of the correlation can be based on the subject similarity between archives and quantified using algorithms such as cosine similarity. This network not only helps to intuitively display the relationship between archives, but also reveals the key nodes and association patterns in the archive collection through network analysis technology.

[0076] Step 4. Construct an archive association network: Using graph theory principles, store the processed archive data in Neo4j, and construct an association network of the processed archives through Neo4j. The steps for constructing the association network are as follows: S401. Design graph model: Design a model of the graph database; S402. Import data: Import the processed archive information into the Neo4j database and create archive nodes; S403. Construct associations: Based on the similarity and subject association between the processed archives obtained by the hierarchical LDA model analysis, construct the relationship between archives. The nodes in the graph database represent the processed archives, and the edges represent the association relationship between the processed archives. The attributes of each node include archive ID, title and date, and the attributes of each edge are the strength of the association.

[0077] Neo4j is a high-performance NoSQL graph database that stores and queries data in a graphical manner. It is very suitable for representing complex associations, such as subject-based relationships between archives. When building associations, if archive A and archive B have a high degree of similarity, archives A and B can be associated. Here, the line between the two archives represents the association between the two, and the attributes of the two lines store the strength or similarity of the association.

[0078] Step 5. Search application: Users search for specific files based on keywords through the search interface, and the system will intelligently recommend other files related to the searched files.

[0079] This intelligent recommendation-based approach not only improves retrieval efficiency, but also may reveal relevant information that users have not noticed.

[0080] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope of the principles and spirit disclosed in this application.

Claims

1. A file association method based on subject association relationship analysis, characterized in that: The following steps are involved: S1. Data preparation and preprocessing: Collect the archival data to be analyzed, preprocess the archival data through natural language processing technology, and then conduct in-depth analysis to obtain the processed archives; S2. Model building: applying the LDA algorithm to model the potential topics in the processed archive collection and building a hierarchical LDA model; S3, file association: based on the topic distribution output by the hierarchical LDA model, the similarity between the processed files is calculated, thereby realizing automatic association of the processed files; S4, constructing an archive association network: using graph theory principles, storing the processed archive data in Neo4j, and constructing an association network of the processed archives through Neo4j; S5. Search application: Users search for specific files based on keywords through the search interface, and the system will intelligently recommend other files related to the searched files.

2. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S1, the collected archival data includes archival metadata and original text files.

3. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S1, the archival data preprocessing includes the following steps: S101, text cleaning: removing non-text elements in the original text file; S102, word segmentation processing: decomposing the original text file content; S103, stop word removal: deleting common words in the original text file that have no actual analysis value; S104, stem extraction: restore the word to its root form to reduce the impact of word form changes on the analysis; S105. Build professional dictionaries: Establish professional dictionaries in specific fields.

4. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S102, the original text file content is decomposed into a set of words and phrases, and Chinese word segmentation is performed on the Chinese content.

5. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S2, the hierarchical LDA model processes conventional vocabulary and professional terms respectively, integrates the domain knowledge provided by professional terms, reveals the deep-level connections between different processed archives based on themes, goes beyond simple keyword matching, and achieves more accurate and comprehensive associations.

6. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S2, the training operation steps of the hierarchical LDA model are as follows: S201, parameter setting: determining key parameters of the hierarchical LDA model; S202, model training: using the processed archive to train a hierarchical LDA model; S203, model evaluation and iteration: evaluating the performance of the hierarchical LDA model to ensure that the model can accurately capture the topics in the text, and making repeated corresponding adjustments based on the evaluation results to achieve optimal performance; S204, topic identification: After the training is completed, each topic is associated with a group of words representing the topic content. At the same time, each of the processed files is assigned a topic distribution to show the degree of association.

7. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S3, the specific method of automatic association is as follows: S301, similarity calculation: using the two topic distributions generated by the hierarchical LDA model of the processed archives, using cosine similarity and Jaccard similarity measurement methods, calculate the similarity between the processed archives; S302, file association: according to the similarity results between the processed files, the processed files with high similarity are associated with each other.

8. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S301, the similarity result between the processed archives is: sim=α*sim1+β*sim2, sim represents the archive similarity result, sim1 represents the similarity result calculated based on the topic distribution of the conventional LDA model, sim2 represents the similarity result calculated based on the topic distribution of the professional LDA model, and α and β are the similarity ratios of sim1 and sim2.

9. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S4, the steps of constructing the association network are as follows: S401, design graph model: design a model of a graph database; S402, importing data: importing the processed archive information into the Neo4j database to create an archive node; S403, building associations: building relationships between archives based on the similarities and subject associations between the processed archives obtained through the hierarchical LDA model analysis.

10. The method for associating archives based on subject association analysis according to claim 1, characterized in that: In step S401, the nodes in the graph in the graph database represent processed archives, and the edges represent the association relationship between processed archives. The attributes of each node include archive ID, title and date, and the attributes of each edge are the strength of association.

Images