Automatic archive editing and research system based on machine learning and natural language processing

Abstract

The invention discloses an archive editing and research automation system based on machine learning and natural language processing, and relates to the technical field of archive management and information processing, and the system comprises an archive editing and research system which automatically collects, arranges, analyzes, studies and compiles archive information to form various archive editing and research results, and a data collection and processing module which is used for collecting and processing the archive information. The method is used for collecting archive information from various channels. According to the archive editing and research automation system based on machine learning and natural language processing, through the data collecting and processing module and the natural language processing module, information can be automatically extracted, the time cost and the error rate of traditional manual retrieval are greatly reduced, the efficiency and the accuracy of processing a large amount of diversified and unstructured archive data are improved, and the archive editing and research automation system is suitable for popularization and application. Through the machine learning module, the efficiency and systematicness of data management are improved by utilizing the classification and arrangement functions of machine learning, and deep knowledge is conveniently extracted from archive data through the application of the NLP technology of the archive editing and research generation module.

Description

Technical Field

[0001] The present invention relates to the field of archive management and information processing technology, and in particular to an automated archive compilation and research system based on machine learning and natural language processing. Background Art

[0002] Archives compilation and research is a very important part of archives work. Editors and researchers can not only collect consultation on related events offline, but also query relevant archive information through the online archive system to complete the compilation and research work to form yearbooks, special topics and other important documents, so as to better serve the society.

[0003] Archival compilation and research work itself is also a complex process involving the analysis, archiving and utilization of a large amount of text data. Traditionally, this process is highly dependent on manual operations, including the reading, understanding, classification, compilation and research of texts. These operations are not only time-consuming and labor-intensive, but also easily affected by human resource limitations, subjective judgment bias and information processing capacity limitations, resulting in inefficient archive utilization and unstable research quality.

[0004] When editorial and research personnel obtain archival information for compilation, the amount of archival information data is huge, and it is more difficult to manually process massive information and extract the content required for editing and research. Archival information covers a variety of formats such as text, images, audio, video, etc., and traditional methods are difficult to efficiently integrate and analyze. Archival texts are mostly unstructured information, which requires an in-depth understanding of the content to extract. Manual operations are inefficient and easy to miss key information. The research field is constantly developing, and archival information needs to be frequently updated and reviewed. Traditional editing and research methods are difficult to adapt quickly, and easily cause content to become outdated. Summary of the invention

[0005] The purpose of the invention provided by the present invention is to provide an automated archive compilation and research system based on machine learning and natural language processing to solve the problems in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an automated archive compilation and research system based on machine learning and natural language processing, comprising: The archive compilation and research system automatically collects, organizes, analyzes, studies and compiles archive information to form various archive compilation and research results; The data collection and processing module is used to collect archival information from various channels, including digital scanning of paper archives, importing of electronic archives and capturing of network information, and to clean, classify, annotate and store the collected archival data for subsequent analysis and processing; The natural language processing module performs linguistic analysis on text data, including lexical and syntactic analysis, entity recognition, extraction of key entities such as people, places, and events, semantic analysis, understanding of text meaning, and mining of semantic relationships between words and sentences; The machine learning module is trained based on a large amount of data and builds models through data preparation, feature engineering, feature selection, data training, and model evaluation steps; The archive compilation and research generation module builds a knowledge graph, uses data mining and artificial intelligence technology to conduct in-depth analysis, and generates a compilation and research report containing archive information summary, in-depth analysis and insights based on the analysis results; The user interface module provides a user query interface, supports users to input query conditions, collects user feedback on research results, and uses feedback to optimize models and algorithms.

[0007] The archive compilation and research system includes a data analysis module, a data encryption module and a data storage module, and the data analysis module, the data encryption module and the data storage module are all connected by bidirectional signals. The data acquisition and processing module includes a paper archive processing module, an online resource crawling module and a data processing module, and the paper archive processing module, the online resource crawling module and the data processing module are connected by bidirectional signals. The machine learning module includes a text training module and a model evaluation module, and the text training module and the model evaluation module are all connected by bidirectional signals.

[0008] Furthermore, a bidirectional signal connection is established between the data collection and processing module and the archive compilation and research system.

[0009] Furthermore, there is a bidirectional signal connection between the machine learning module and the archive compilation and research system.

[0010] Furthermore, a bidirectional signal connection is established between the natural language processing module and the archive compilation and research system.

[0011] Furthermore, a bidirectional signal connection is established between the data acquisition and processing module and the natural language processing module.

[0012] Furthermore, there is a bidirectional signal connection between the natural language processing module and the machine learning module.

[0013] Furthermore, the archive compilation and research generation module includes a knowledge extraction module and a compilation and research analysis module, and a bidirectional signal connection is established between the knowledge extraction module and the compilation and research analysis module.

[0014] Furthermore, the user interface module includes an interface query module, a feedback module, an information retrieval module and a report generation module, and the interface query module, feedback module, information retrieval module and report generation module are all bidirectionally signal-connected.

[0015] Furthermore, a bidirectional signal connection is established between the archive compilation and research generation module and the archive compilation and research system.

[0016] Furthermore, a bidirectional signal connection is established between the user interface module and the archive compilation and research system.

[0017] The present invention provides an automated archive compilation and research system based on machine learning and natural language processing. It has the following beneficial effects: This automated archive compilation and research system based on machine learning and natural language processing can automatically extract information through data collection and processing modules and natural language processing modules, greatly reducing the time cost and error rate of traditional manual retrieval, and improving the efficiency and accuracy of processing large amounts of diverse, unstructured archival data. Through the machine learning module, the classification and organization functions of machine learning are utilized to improve the efficiency and systematicness of data management. Through the application of NLP technology in the archive compilation and research generation module, it is convenient to extract deep knowledge from archival materials, thereby improving the quality and depth of archive compilation and research. It can automatically generate compilation and research reports, reducing the workload of researchers.

[0018] This automated archive compilation and research system based on machine learning and natural language processing can automatically process and improve the efficiency of obtaining archival information through the collaborative work of various modules, optimize the classification, integration and analysis of information, promote the in-depth utilization of archival materials, and realize the automation of archive compilation and research, thereby solving the problems of inefficiency and insufficient information utilization in the traditional archive compilation and research process. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a system overall architecture diagram of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Figure 2 This is the overall system diagram of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Figure 3 A schematic diagram of an archive compilation and research system of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Figure 4 A schematic diagram of a data collection and processing module of an automated archive compilation and research system based on machine learning and natural language processing according to the present invention; Figure 5 A schematic diagram of a machine learning module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Figure 6 A schematic diagram of an archive compilation and research generation module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Figure 7 A schematic diagram of a user interface module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Figure 8It is a flow chart of the data collection and processing module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Fig. 9 A flowchart of a natural language processing module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Fig.10 A flowchart of a machine learning module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention; Fig.11 This is a flow chart of the archive compilation and research generation module of the archive compilation and research automation system based on machine learning and natural language processing of the present invention.

[0020] In the figure: 1. Archive compilation and research system; 2. Data collection and processing module; 3. Natural language processing module; 4. Machine learning module; 5. Archive compilation and research generation module; 6. User interface module. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0022] Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0023] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: See also Figure 1-11 The present invention provides a technical solution: an automated archive compilation and research system based on machine learning and natural language processing, comprising: Archives compilation and research system 1, automatically collects, organizes, analyzes, studies and compiles archive information to form various archives compilation and research results; Data collection and processing module 2 is used to collect archival information from various channels, including digital scanning of paper archives, import of electronic archives and capture of network information, and to clean, classify, annotate and store the collected archival data for subsequent analysis and processing; Natural language processing module 3 performs linguistic analysis on text data, including lexical and syntactic analysis, entity recognition, extraction of key entities such as people, places and events, semantic analysis, understanding of text meaning and mining semantic relationships between words and sentences; Machine learning module 4, training based on large amounts of data, building models through data preparation, feature engineering, feature selection, data training, and model evaluation steps; Archives compilation and research generation module 5, builds a knowledge graph, uses data mining and artificial intelligence technology to conduct in-depth analysis, and generates a compilation and research report containing archive information summary, in-depth analysis and insights based on the analysis results; User interface module 6 provides a user query interface, supports users to input query conditions, collects user feedback on research results, and uses feedback to optimize models and algorithms.

[0024] The archive compilation and research system 1 includes a data analysis module, a data encryption module and a data storage module. The data analysis module, the data encryption module and the data storage module are all connected by two-way signals. The data acquisition and processing module 2 includes a paper archive processing module, an online resource crawling module and a data processing module. The paper archive processing module, the online resource crawling module and the data processing module are connected by two-way signals. The machine learning module 4 includes a text training module and a model evaluation module. The text training module and the model evaluation module are all connected by two-way signals. The data analysis module analyzes the data. The data encryption module includes data transmission encryption and data storage encryption. Data transmission encryption uses the SSL / TLS protocol to encrypt the data transmission in the system to ensure data Security during transmission, data storage encryption is to encrypt sensitive data stored in the database to prevent unauthorized access. The data storage module is responsible for efficient storage and management of processed data for easy and fast retrieval. The paper archive processing module uses a high-resolution scanner to convert paper archives into electronic images, and then uses OCR technology to convert the text in the image into editable text. OCR technology can use Tesseract tools to ensure high-precision text recognition, and use image recognition technology to automatically annotate and extract key information in the image, such as the subject of the image or the time of shooting. The online resource crawling module uses the crawler BeautifulSoup tool to crawl on a specified website or platform according to preset rules. To capture relevant data, the web crawler uses the Scrapy framework and combines it with Selenium to process dynamic web pages to ensure that it can capture content rendered by JavaScript. The Tesseract OCR engine is used in combination with deep learning models such as CRNN to improve the accuracy of text recognition. The Google Speech-to-Text API is used to convert the voice content in audio and video into text to ensure the comprehensive collection of archival information. Through natural language processing technologies such as keyword matching and topic model LDA, the content related to the editing and research topics is preliminarily screened out to ensure the relevance of the data. The data processing module includes data cleaning and data classification and annotation. Data cleaning uses regular expressions and stop word lists. These technologies remove duplicate data and noise, and automatically correct format errors in the data, such as date format or name spelling, through rule engines or machine learning models. Data classification and annotation uses the DBSCAN clustering algorithm to classify data, group them by age and theme, and use named entity recognition (NER) technology to mark key entities in the data, such as people, places or events, to facilitate subsequent knowledge extraction and information retrieval. The text training module uses the BERT model to pre-train text in text classification tasks, learn the semantic representation of text, and then fine-tune it for archival data classification to improve classification accuracy. In image recognition tasks, convolutional neural networks (CNNs) are used for image classification and key information extraction.In speech recognition tasks, recurrent neural network (RNN) or Transformer models are used to improve the accuracy of speech-to-text conversion and achieve deep learning. The model evaluation module evaluates the model performance through multiple indicators to determine whether it meets the requirements.

[0025] Specifically, there is a two-way signal connection between the data acquisition and processing module 2 and the archive compilation and research system 1. The data acquisition and processing module 2 determines the data source, covering types such as paper archives, electronic files and online resources, to ensure comprehensive collection of archival information, use crawler technology to obtain data, and use scanning technology to process paper archive data. Various types of data are collected, and data are screened according to preset conditions to filter out irrelevant data, improve data quality, and standardize data, converting data in different formats into a unified format for subsequent processing. Finally, the data storage saves the processed data to the database to provide data support for other modules of the system.

[0026] Specifically, there is a bidirectional signal connection between the machine learning module 4 and the archive compilation and research system 1. In the data preparation stage, the machine learning module 4 collects and organizes data for training the model. Feature engineering extracts data features and converts the original data into a form that the model can understand. Feature selection selects the most representative features to improve the efficiency and accuracy of model training. Data training uses the selected machine learning algorithm to train the data and build a model. The model performance is evaluated through multiple indicators to determine whether it meets the requirements. If the performance meets the requirements, the model is deployed and the trained model is applied to the actual system to realize data processing and analysis functions.

[0027] Specifically, there is a two-way signal connection between the natural language processing module 3 and the archive compilation and research system 1. After the text data is input, the natural language processing module 3 performs linguistic analysis on the text, such as lexical and syntactic analysis, to lay the foundation for subsequent processing. Entity recognition extracts key entities such as people, places, and events from the text, and semantic analysis understands the meaning of the text and mines the semantic relationship between words and sentences. Finally, the entity recognition results and semantic analysis results are output to provide key information for knowledge extraction and data organization.

[0028] Specifically, a bidirectional signal connection is established between the data acquisition and processing module 2 and the natural language processing module 3 , and the natural language processing module 3 further processes and analyzes the data processed by the data acquisition and processing module 2 .

[0029] Specifically, there is a bidirectional signal connection between the natural language processing module 3 and the machine learning module 4 , and the machine learning module 4 optimizes and integrates the information in the natural language processing module 3 .

[0030] Specifically, the archive compilation and research generation module 5 includes a knowledge extraction module and a compilation and research analysis module. There is a bidirectional signal connection between the knowledge extraction module and the compilation and research analysis module. The knowledge extraction module extracts relevant entities and their relationships from the processed data through NER and relationship extraction technology, constructs a knowledge graph, uses the BERT-basedNER model to identify various entities in the text, such as names of people, places or organizations, uses the BERT-based relationship extraction model to determine the relationship between entities, such as causal relationships and subordinate relationships, uses rule-based or machine learning classification algorithms to classify entities into corresponding concept categories, and continuously enriches and updates the knowledge graph. Use data analysis algorithms, such as trend analysis, association analysis, and sentiment analysis models, to conduct in-depth analysis of archival data. Trend analysis uses a time series analysis algorithm to analyze the development trend of a certain field in the archives. Association analysis uses the Apriori association rule mining algorithm to find the potential connections between different events and characters. Sentiment analysis uses a BERT-based sentiment analysis model to determine the sentiment tendency of the archival text towards a certain event or policy, such as positive, negative, and neutral, to provide a more comprehensive analysis perspective for the compilation and research report. Based on the analysis results, the system can provide targeted recommendation strategies for archive management and research, such as recommending relevant research directions and policy recommendations.

[0031] Specifically, the user interface module 6 includes an interface query module, a feedback module, an information retrieval module and a report generation module. The interface query module, the feedback module, the information retrieval module and the report generation module are all bidirectionally connected by signals. The interface query module can provide a friendly user interface and support users to enter keywords and time ranges for query. Elasticsearch search engine technology can be used to ensure rapid response to queries. The feedback module allows users to evaluate the research results and make modification suggestions. The system collects these feedbacks and uses reinforcement learning or online learning algorithms to optimize models and algorithms to improve the accuracy and practicality of the system. The information retrieval module helps users accurately obtain the required information. The report generation module presents the final research results in the form of a report.

[0032] Specifically, there is a two-way signal connection between the archive compilation and research generation module 5 and the archive compilation and research system 1. The archive compilation and research generation module 5 uses data mining and artificial intelligence technology to perform in-depth analysis based on the knowledge graph constructed by the knowledge extraction module. For example, similar themes in archival data are discovered through cluster analysis, and sentiment analysis is used to determine the sentiment tendency of related events. Based on the analysis results, a compilation and research report containing archival information summary, in-depth analysis and insights is generated, providing strong support for archive management and research.

[0033] Specifically, a two-way signal connection is established between the user interface module 6 and the archive compilation and research system 1 , through which the user interface module 6 is used to facilitate the user to query the required information.

[0034] During use, the system administrator defines the editing and research topics and related keywords through the user interface in the user interface module 6. The system automatically generates data collection tasks based on these keywords, collects archival information from various channels through the data collection and processing module 2, and cleans, classifies, annotates and stores it. The natural language processing module 3 performs linguistic analysis and entity recognition on the text data to extract key information. The machine learning module 4 trains, builds and optimizes models based on the processed data. The archival editing and research generation module 5 extracts relevant entities and their relationships from the processed data through NER and relationship extraction technology, and constructs a knowledge graph. Based on the knowledge graph, data analysis algorithms such as cluster analysis and trend analysis are used to conduct in-depth analysis of the information in the knowledge graph to generate editing and research reports and recommendation strategies. The system automatically generates an editing and research report that contains archival information summary, in-depth analysis and insights, and presents it to the user through the user interface in the user interface module 6. The user can evaluate the editing and research results and make modification suggestions, and the system adjusts and optimizes according to the feedback.

[0035] The above are only preferred embodiments of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. The automated archive compilation and research system based on machine learning and natural language processing is characterized by: include: Archives compilation and research system (1) automatically collects, organizes, analyzes, studies and compiles archive information to form various archives compilation and research results; The data collection and processing module (2) is used to collect archival information from various channels, including digital scanning of paper archives, importing of electronic archives and capturing of network information, and to clean, classify, annotate and store the collected archival data for subsequent analysis and processing; Natural language processing module (3) performs linguistic analysis on text data, including lexical and syntactic analysis, entity recognition, extraction of key entities such as people, places and events, semantic analysis, understanding the meaning of text and mining the semantic relationship between words and sentences; The machine learning module (4) is trained based on a large amount of data and builds a model through data preparation, feature engineering, feature selection, data training, and model evaluation steps; The archive compilation and research generation module (5) constructs a knowledge graph, uses data mining and artificial intelligence technology to conduct in-depth analysis, and generates a compilation and research report containing archive information summary, in-depth analysis and insights based on the analysis results; A user interface module (6) provides a user query interface, supports users to input query conditions, collects user feedback on the compilation and research results, and uses the feedback to optimize the model and algorithm; The archive compilation and research system (1) comprises a data analysis module, a data encryption module and a data storage module, and the data analysis module, the data encryption module and the data storage module are all bidirectionally connected by signals. The data acquisition and processing module (2) comprises a paper archive processing module, an online resource crawling module and a data processing module, and the paper archive processing module, the online resource crawling module and the data processing module are bidirectionally connected by signals. The machine learning module (4) comprises a text training module and a model evaluation module, and the text training module and the model evaluation module are all bidirectionally connected by signals.

2. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A bidirectional signal connection is established between the data acquisition and processing module (2) and the archive compilation and research system (1).

3. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A two-way signal connection is established between the machine learning module (4) and the archive compilation and research system (1).

4. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A bidirectional signal connection is established between the natural language processing module (3) and the archive compilation and research system (1).

5. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A bidirectional signal connection is established between the data acquisition and processing module (2) and the natural language processing module (3).

6. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A bidirectional signal connection is established between the natural language processing module (3) and the machine learning module (4).

7. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: The archive compilation and research generation module (5) comprises a knowledge extraction module and a compilation and research analysis module, and a bidirectional signal connection is established between the knowledge extraction module and the compilation and research analysis module.

8. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: The user interface module (6) comprises an interface query module, a feedback module, an information retrieval module and a report generation module, and the interface query module, the feedback module, the information retrieval module and the report generation module are all bidirectionally signal-connected.

9. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A bidirectional signal connection is established between the archive compilation and research generation module (5) and the archive compilation and research system (1).

10. The automated archive compilation and research system based on machine learning and natural language processing according to claim 1, characterized in that: A bidirectional signal connection is established between the user interface module (6) and the archive compilation and research system (1).

Images