Intelligent data information rapid checking and question-answering system

Abstract

The invention discloses an intelligent data information rapid checking and question-answering system, which relates to the technical field of intelligent data information and comprises a document import module, a vectorization analysis module, an NLP processing module, a question-answering matching module and a result output module. The document import module is used for receiving and importing documents including but not limited to PDF, DOC, TXT and PPT, analyzing the documents in different formats by adopting corresponding libraries and tools, identifying and classifying the files by identifying title formats and question number formats when processing the files in test questions and test paper formats, and performing preliminary analysis on the contents of the documents by utilizing a machine learning algorithm; according to the invention, the system not only can realize the rapid checking and question-answering functions of documents, but also can be applied to the field of educational training, realizes the online editing, management and test paper formation of test questions, reduces the manpower consumption in basic training business, and integrally improves the working quality and efficiency; meanwhile, the system can help enterprises to carry out internal training and examination, and improves the practicability and application range of the system.

Description

Technical Field

[0001] The present invention relates to the field of intelligent data information technology, and in particular to an intelligent data information rapid inspection and question-answering system. Background Art

[0002] With the acceleration of enterprise intelligent transformation, the demand for efficient data information processing and accurate question-answering systems is becoming increasingly urgent. In the current enterprise environment, the existing information processing methods have many problems and are difficult to meet the needs of enterprise development.

[0003] The prior art has the following problems:

[0004] 1. Single document processing function: The current system has limited capabilities in document processing and can only perform simple information document quick check and question-and-answer. In the field of education and training, it cannot effectively identify, classify, edit online, manage, and compile test questions and test paper format files, resulting in high manpower consumption and low work efficiency in basic training business.

[0005] 2. Insufficient support for education and training: In the field of education and training, there is a lack of effective system support, which makes it impossible to realize the online editing, management and composition of test questions, and cannot meet the needs of teachers to quickly generate test papers that meet teaching needs. It is also impossible to provide students with personalized learning feedback, which affects the quality and efficiency of education and training.

[0006] 3. Limited internal training and assessment: The existing system cannot provide effective support for internal training and assessment. It cannot generate customized training papers and assessment content according to the company's own business needs, making it difficult to improve employees' professional quality and business capabilities, thus limiting the company's development.

[0007] In this context, there is an urgent need to develop an intelligent data information rapid inspection and question-answering system that aims to be able to comprehensively process documents in various formats, use vector databases and natural language processing technology to achieve efficient information retrieval and accurate answers, and perfectly integrate with customer service robots to provide more intelligent services. At the same time, it plays an important role in education and training and internal training assessment in enterprises to meet the needs of enterprise intelligent transformation. Summary of the invention

[0008] The purpose of the present invention is to provide an intelligent data information rapid inspection and question-answering system to solve the problems raised in the above-mentioned background technology.

[0009] In order to solve the above technical problems, the present invention provides an intelligent data information rapid inspection and question-answering system, including a document import module, a vectorization analysis module, an NLP processing module, a question-answer matching module and a result output module;

[0010] Document import module: used to receive and import documents including but not limited to PDF, DOC, TXT, and PPT. It uses corresponding libraries and tools to parse documents of different formats. When processing test questions and test paper format files, it identifies and classifies the files by identifying the title format and question number format. It also uses machine learning algorithms to conduct preliminary analysis of the document content and extract subject keywords for subsequent retrieval and processing.

[0011] Vectorization analysis module: Use text vectorization technology to pre-process the imported document content and then input it into the Transformer model to obtain the vector representation of each word or sentence. Use one of the TF-IDF algorithm and TextRank algorithm to extract keywords, sentences, and paragraphs, and map them to points in the high-dimensional vector space to identify and summarize the key information and structure of the document.

[0012] NLP processing module: Using natural language processing technology, the vectorized document content is subjected to part-of-speech tagging, named entity recognition, and syntactic analysis to obtain vocabulary, grammar, and context information, identify the document theme, key information, and structure, use the topic model for topic identification, and save the document content according to the preset classification system, establish a label system, and classify it into corresponding categories for subsequent retrieval and matching;

[0013] Question-answer matching module: pre-process the text, then use the pre-trained Word2Vec model to convert the question words into vectors, then get the question vector representation by averaging and summing, and use TF-IDF combined with the bag-of-words model to calculate the TF-IDF value of each word in the question to build a bag-of-words vector; and use cosine similarity and Euclidean distance in the vector space; perform the same vectorization processing on each document in the document library as the question, and calculate and store the document vector when the document is imported; use dependency syntax analysis in natural language processing technology to analyze the semantic structure of the question, determine the key entities and attributes of the question, and search for content with the same or similar entities and attributes in the document; use the topic model LatentDirectAllocation to extract the document topic, and consider the relevance of the question topic and the document topic when matching the question. If the topics are similar, the document is more likely to be the answer to the question; for test questions and analysis, the module associates them based on keywords and key steps; through TF-IDF, the module associates them based on keywords and key steps; The DF keyword extraction algorithm extracts keywords from the test questions and analysis documents respectively, and then uses similarity algorithms such as cosine similarity to calculate the similarity between keywords based on the matching degree of the keywords, and determines the degree of correlation between the test questions and the analysis;

[0014] Result output module: organizes the matching results into paragraphs, organizes them reasonably according to question type and document structure, extracts relevant sentences and paragraphs to organize answers, converts them into human language using language models and returns them to users, controls the output style and accuracy by adjusting language model parameters, supports test paper generation and formatted output, and is used to customize the output test paper format and content according to user needs.

[0015] Furthermore, the document import module includes a user interface design unit, a document format conversion unit, a batch import unit, and a document storage and management unit.

[0016] Furthermore, in the document import module, a mechanism for continuously updating the annotation data is established; as new documents are continuously imported and the system is used, new documents are manually annotated regularly and the new annotation data is added to the training set; at the same time, the annotated documents are reviewed and updated.

[0017] Furthermore, in the result output module, the system also provides multi-angle information display.

[0018] Furthermore, the document import module also includes a preview and editing unit, an automatic format detection and conversion unit, an error handling and log recording unit, and a permission management and security unit.

[0019] Furthermore, the vectorized analysis module also includes a spelling correction unit, a synonym replacement unit, a sentiment analysis unit, a topic model unit, a named entity recognition unit, a context feature unit, a hybrid embedding unit, a dynamic embedding unit, a dimension optimization unit, a semantic similarity measurement unit, a clustering and classification unit, a visualization and explanatory unit, and a modular design unit.

[0020] Furthermore, the dimension optimization unit includes an adaptive dimension adjustment subunit, a task-based dimension learning subunit, a dimension sharing and selection subunit, a sparsification subunit, and a knowledge distillation subunit.

[0021] Furthermore, the NLP processing module introduces an opinion mining mechanism in the process of identifying and understanding sentiment tendencies and opinions in texts, and introduces time-series-based sentiment trend analysis, multi-dimensional fine-grained opinion mining, and social network analysis and sentiment propagation models in the basic implementation steps of the opinion mining mechanism; time-series-based sentiment trend analysis includes the introduction of time series models and sentiment fluctuation measurements, multi-dimensional fine-grained opinion mining includes the construction of aspect dictionaries and opinion associations based on knowledge graphs, and social network analysis and sentiment propagation models include user influence evaluation and sentiment propagation modeling.

[0022] Furthermore, in the question-answer matching module, a pre-trained language model is introduced and combined with a knowledge graph; and personalization and context awareness are introduced to add personalized elements to question-answer matching based on the user's personal preferences and historical behavior data, as well as by introducing a technique in logical reasoning or causal reasoning.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] This system can not only realize the rapid checking and question-answering functions of documents, but can also be applied to the field of education and training to realize the online editing, management and compilation of test questions, reduce the manpower consumption in basic training business, and improve the overall work quality and efficiency; at the same time, it can help enterprises conduct internal training and assessment, improving the practicality and application scope of the system.

[0025] By establishing a mechanism for continuously updating annotated data, this intelligent data information rapid inspection and question-answering system can better adapt to the ever-changing document content and user needs in the education field, and provide more accurate and useful information services for educators and students.

[0026] By organizing or scanning the company's existing information into documents and importing them into the company's information database, the system will automatically analyze and quantify the internal information, classify and save the information content through NPL, understand the questions when the customer needs it, and return matching answers. Therefore, customer service does not need manual intervention, saving costs and improving experience. It also provides 24 / 7 service capabilities without time constraints, and improves customer service efficiency and quality, facilitates unified summarization of corporate information, and facilitates internal personnel to search by asking questions, thereby accelerating the company's intelligent transformation. At the same time, the permissions of different knowledge bases are differentiated, and each department focuses on its own resources, enhancing the accessibility and sharing of corporate knowledge.

[0027] Enterprises can efficiently import various documents into the intelligent data information rapid inspection and question-and-answer system, providing a rich data source for subsequent data analysis and question-and-answer functions. At the same time, a multi-scale or hierarchical vector representation method is constructed, which can not only capture multiple levels of information in the text, but also flexibly choose which dimensions to use according to task requirements, achieving a balance between flexibility and efficiency. The question-and-answer matching module finds the most relevant answers in the knowledge base or document set, and organizes them into easy-to-understand paragraphs, helping users quickly obtain accurate and useful information, improving user experience and satisfaction, monitoring and analyzing real-time changes in opinions on social media, mining fine-grained opinion information, and revealing the propagation mechanism and influencing factors of emotional tendencies, which is of great significance for corporate decision-making, product improvement and public opinion response. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 The present invention is a flowchart of an intelligent data information rapid inspection and question-answering system. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1 , the present invention provides a technical solution:

[0031] See also Figure 1 As shown, an intelligent data information rapid inspection and question-answering system includes a document import module, a vectorization analysis module, an NLP processing module, a question-answering matching module and a result output module;

[0032] Document import module: used to receive and import documents including but not limited to PDF, DOC, TXT, and PPT. It uses corresponding libraries and tools to parse documents of different formats. When processing test questions and test paper format files, it identifies and classifies the files by identifying the title format and question number format. It also uses machine learning algorithms to conduct preliminary analysis of the document content and extract subject keywords for subsequent retrieval and processing.

[0033] Vectorization analysis module: Use text vectorization technology to pre-process the imported document content and then input it into the Transformer model to obtain the vector representation of each word or sentence. Use one of the TF-IDF algorithm and TextRank algorithm to extract keywords, sentences, and paragraphs, and map them to points in the high-dimensional vector space to identify and summarize the key information and structure of the document.

[0034] NLP processing module: Using natural language processing technology, the vectorized document content is subjected to part-of-speech tagging, named entity recognition, and syntactic analysis to obtain vocabulary, grammar, and context information, identify the document theme, key information, and structure, use the topic model for topic identification, and save the document content according to the preset classification system, establish a label system, and classify it into corresponding categories for subsequent retrieval and matching;

[0035] Question-answer matching module: When receiving questions from users, the text is first preprocessed to remove irrelevant content in the question, including punctuation marks and special characters. For English questions, stem extraction and part-of-speech restoration are also performed to convert them into basic forms. Then, the question words are converted into vectors using the pre-trained Word2Vec model, and the question vector representation is obtained by averaging or summing. The TF-IDF value of each word in the question is calculated using the bag-of-words model to construct the bag-of-words vector. Cosine similarity and Euclidean distance are used in the vector space. Each document in the document library is also vectorized in the same way as the question, and the document vector is calculated and stored when the document is imported. The cosine similarity value ranges from -1 to 1, and the closer it is to 1, the more similar the two vectors are; the smaller the Euclidean distance, the closer the two vectors are. The semantic structure of the question is analyzed with the help of dependency syntax analysis in natural language processing technology, the key entities and attributes of the question are determined, and the content with the same or similar entities and attributes is searched in the document. The topic model LatentDirectAllocat on extracts document topics. When matching questions, consider the relevance between the topic of the question and the topic of the document. If the topics are similar, the document is more likely to be the answer to the question. For test questions and solutions, the module associates them based on keywords and key steps. The TF-IDF keyword extraction algorithm is used to extract keywords from test questions and solution documents. For test questions, keywords such as "equation", "solution", "quadratic equation", etc. can be extracted. For solution documents, descriptions of key steps such as "factorization" and "root-finding formula" can be extracted. Then, based on the matching degree of keywords, similarity algorithms such as cosine similarity are used to calculate the similarity between keywords to determine the degree of association between test questions and solutions.

[0036] Result output module: organizes the matching results into paragraphs, organizes them reasonably according to the question type and document structure, extracts relevant sentences or paragraphs to organize the answers, converts them into human language using the language model and returns them to the user. It controls the output style and accuracy by adjusting the language model parameters or using specific prompt words, supports test paper generation and formatted output, and is used to customize the output test paper format and content according to user needs;

[0037] Example description:

[0038] Document import module:

[0039] Suppose we have a PDF file of a math test paper and a DOC file of a history research report that need to be imported into the system.

[0040] Format parsing: For PDF files, the system uses the PyPDF2 library to read the document content. Extract text information page by page. If there are charts in the test paper, you may also need to use OCR technology to extract text from the pictures. For DOC files, use the python-docx library to extract paragraph text, titles, and other content.

[0041] Identification and classification of title format and question number format: For example, in a math test paper, the title may be "Mathematics Final Test", and the question number format is "1." "2.", etc. The system classifies the document as a math test paper by identifying these features. In a historical research report, the title may be "Research on Social Changes in a Certain Historical Period", and there is no obvious question number format. The system classifies it as a research report based on these features.

[0042] Subject keyword extraction: For math test papers, machine learning algorithms may extract subject keywords such as "mathematics", "function", "geometry", etc. For historical research reports, they may extract keywords such as "history", "social change", "specific historical period", etc.

[0043] Vectorized analysis module:

[0044] Take a TXT file of a science and technology news as an example:

[0045] Text preprocessing: remove punctuation and special characters, such as changing "This is a news article about artificial intelligence." to "This is a news article about artificial intelligence."

[0046] Input to Transformer model: The preprocessed text is input to the Transformer model to obtain a vector representation of each word. For example, the word "artificial intelligence" is represented as a vector of a specific dimension.

[0047] Keyword, sentence, and paragraph extraction: Use the TF-IDF algorithm to calculate the TF-IDF value of each word in the document. Assuming that the TF-IDF values ​​of words such as "artificial intelligence", "machine learning", and "deep learning" are high, these words are extracted as keywords. At the same time, based on the distribution of keywords, key sentences and paragraphs can be extracted. For example, a paragraph containing multiple keywords is considered a key paragraph.

[0048] NLP processing module:

[0049] Take the PPT file of a novel as an example:

[0050] Part-of-speech tagging, named entity recognition, and syntactic analysis: Perform part-of-speech tagging on the text content in the PPT to determine the part of speech of each word, such as "noun", "verb", "adjective", etc. Perform named entity recognition to identify entities such as people, places, and organizations, such as "Harry Potter" and "Hogwarts". Through syntactic analysis, determine the grammatical structure of the sentence, such as "subject + predicate + object", etc.

[0051] Topic identification: Use a topic model (such as LDA) to identify the topics of the novel content. Suppose the identified topics are "magic adventure", "friendship", "growth", etc.

[0052] Classification and preservation and establishment of a label system: According to the preset classification system, the novel is classified into the "Literary Works" category. A label system is established to label the novel with "magic novel" or "adventure story" for subsequent retrieval and matching.

[0053] Question-answer matching module:

[0054] Suppose the user asks "How do I solve a quadratic equation?":

[0055] Text preprocessing: remove punctuation marks in the question and convert it into “How to solve quadratic equations?” For English questions, if it is “How to solve quadratic equations?”, perform stem extraction and part-of-speech restoration, such as restoring “sol ves” to “sol ve”.

[0056] Vectorization: Use the Word2Vec model to convert the words in the question into vectors, and then get the question vector representation by averaging. At the same time, use TF-IDF combined with the bag-of-words model to calculate the TF-IDF value of each word in the question and construct a bag-of-words vector.

[0057] Cosine similarity and Euclidean distance calculation: Perform the same vectorization process on each document in the document library as the question, and calculate and store the document vector when the document is imported. Calculate the cosine similarity and Euclidean distance between the question vector and the document vector. For example, the cosine similarity between the document vector and the question vector of a paper on math problem solving methods is high, but the Euclidean distance is small.

[0058] Semantic structure analysis and topic matching: Through dependency syntax analysis, it is determined that the key entity of the question is "quadratic equation" and the key attribute is "solving". Search for content with the same or similar entities and attributes in the document. Use the topic model LatentDirectAllocation to extract the document topic and find that the topic of the document is related to math problem solving and is similar to the topic of the question.

[0059] Correlation between test questions and solutions: For test question documents, extract keywords such as "quadratic equation" and "solution". For solution documents, extract key step descriptions such as "matching method" and "root-finding formula". Use the TF-IDF keyword extraction algorithm to extract keywords from the questions and match them with the document keywords. Calculate the similarity between keywords, such as cosine similarity, to determine the degree of correlation between the test questions and solutions.

[0060] Result output module:

[0061] Assume the system finds math documents relevant to the user's question:

[0062] Organize answers: Organize matching results into paragraphs, extract relevant sentences or paragraphs to organize answers. For example, find a paragraph about the solution of quadratic equations and organize it into answer content.

[0063] Language model conversion: Use language models to convert answers into human language to make them easier to understand. For example, professional mathematical terms can be converted into easy-to-understand explanations.

[0064] Control output style and accuracy: By adjusting language model parameters or using specific prompt words, you can control the style of the output. For example, you can set it to a concise and clear style or a professional and detailed style. Ensure the accuracy of the output, check and verify the answers, and avoid erroneous information.

[0065] Test paper generation and formatted output: Customize the output test paper format and content according to user needs. For example, if a user asks to generate a math test paper, the system can select relevant test questions and explanations from the document library and format and output them according to the test paper format;

[0066] Effect:

[0067] Efficient document processing: It can receive and parse documents in various formats, including PDF, DOC, TXT, PPT, etc., which provides great convenience for users without worrying about incompatible document formats. For test questions and test paper format files, it can accurately classify them by identifying specific format marks, which improves the efficiency of document management. At the same time, extracting subject keywords helps to quickly retrieve and process documents.

[0068] Accurate information extraction and analysis: The vectorization analysis module uses advanced technology to convert document content into vector representation. Combined with the TF-IDF algorithm and TextRank algorithm, it can accurately extract keywords, sentences and paragraphs, and better identify and summarize the key information and structure of the document. The NLP processing module uses part-of-speech tagging, named entity recognition and syntactic analysis to fully obtain the vocabulary, grammar and context information of the document, accurately identify the document theme, key information and structure, and provide a reliable basis for subsequent retrieval and matching.

[0069] Accurate question-answer matching: After receiving questions from users, the question-answer matching module performs multi-dimensional vector processing and uses metrics such as cosine similarity and Euclidean distance to search for the most matching document vector in the vector space. At the same time, through semantic structure analysis and topic matching, as well as the association processing of test questions and solutions, it can provide users with accurate answers.

[0070] Flexible result output: The result output module can organize the matching results into paragraphs and convert them into human language using the language model to meet the different needs of users. It supports test paper generation and formatted output, and can customize the test paper format and content according to user needs, providing strong support for the fields of education and training.

[0071] Application in education:

[0072] In the field of education and training, the system realizes the online editing, management and composition of test questions, which greatly reduces the manpower consumption in basic training business. Teachers can quickly generate test papers that meet teaching needs, improving the efficiency of teaching work. At the same time, students can conduct independent learning and testing through the system and obtain personalized learning feedback.

[0073] Assistance for internal training and assessment: For enterprises, the system can help with internal training and assessment. Enterprises can customize training papers and assessment content according to their own business needs to improve the professional quality and business capabilities of employees. At the same time, the system's precise question-and-answer function can provide employees with timely knowledge support and improve the overall work quality and efficiency of the enterprise.

[0074] Comprehensive utilization of cross-format documents: The system can process documents in multiple formats, so that information from different sources can be integrated and utilized on one platform. For example, academic papers in PDF format, research reports in DOC format, and training materials in PPT format can be comprehensively analyzed to provide users with more comprehensive knowledge services.

[0075] Ability to handle problems in different languages: For English questions, the system performs stem extraction and part-of-speech restoration to convert them into basic forms, improving the ability to handle problems in different languages. This provides convenience for multinational companies and international educational institutions and promotes cross-cultural knowledge exchange and cooperation.

[0076] Potential for continuous learning and optimization: As the document library continues to grow and user questions accumulate, the system can continue to learn and optimize to improve its accuracy and practicality. For example, by analyzing user feedback, adjusting language model parameters and matching algorithms, the system's performance can be further improved.

[0077] Furthermore, the document import module includes:

[0078] User Interface Design Unit: Design a user interface that allows users to upload and import documents, provide document format preview and format conversion options to ensure that documents can be correctly parsed by the system; allow users to add categories and tags to documents during import to facilitate subsequent retrieval and query; provide automatic recommendation functions for categories and tags, and automatically assign appropriate categories and tags to documents based on their content;

[0079] Document format conversion unit: For documents in non-electronic formats (such as paper documents), provide scanning functions and use one of the external scanning devices to convert them into electronic documents. For documents that are already in electronic format but the format is not uniform or not supported by the system, provide format conversion tools or interfaces to convert them into a format that the system can recognize (such as PDF, DOC, TXT, PPT, etc.); consider supporting multiple languages ​​to ensure that the system can process document content in different languages; provide multi-language interfaces and prompt information to meet the needs of users in different countries and regions;

[0080] Batch import unit: allows users to import multiple documents at one time, providing a progress bar or import status prompt to display the import progress and results; during the import of a large number of documents, the system performs optimization processing and adopts one of the multi-threading and asynchronous processing methods to improve the import speed and efficiency;

[0081] Document storage and management unit: Stores imported documents in the system database to ensure data security and accessibility, provides document management and retrieval functions, and facilitates subsequent editing, querying, and use of documents. For documents that need to be updated frequently, a version control mechanism is introduced to record the change history and import time of each version to facilitate subsequent tracking and management.

[0082] Examples:

[0083] Suppose a technology company needs to import its past project reports, technical documents and product manuals into the intelligent data information quick check and question-answering system. The following is the specific operation process:

[0084] Document organization and scanning:

[0085] The company's document manager first collects paper project reports and product manuals and converts them into electronic documents in PDF format using a scanner. For documents that are already in electronic format (such as Word documents or PPT presentations), the document manager checks whether their formats are uniform and confirms whether they need to be converted to PDF or other formats supported by the system.

[0086] Using the document import interface:

[0087] The document administrator logs in to the user interface of the intelligent data information quick check and question-answering system; in the document import module, selects the "batch import" function, and selects the sorted electronic documents through the file selection dialog box;

[0088] Document format conversion unit and import:

[0089] The system automatically detects the format of the selected document and prompts the user to convert the format of the document that needs to be converted; after the user confirms the conversion, the system starts to import documents in batches and displays the import progress bar on the interface.

[0090] Import completion and subsequent operations:

[0091] When all documents have been imported, the system displays a prompt message indicating that the import is successful and lists the imported documents. The document administrator can further edit the metadata of the documents (such as title, description, category, etc.) to facilitate more accurate retrieval and use of these documents in the future.

[0092] Through such implementation steps and examples, enterprises can efficiently import various documents into the intelligent data information rapid inspection and question-and-answer system, providing a rich data source for subsequent data analysis and question-and-answer functions.

[0093] Furthermore, in the document import module, a mechanism for continuously updating the annotation data is established; with the continuous import of new documents and the use of the system, the new documents are manually annotated regularly, and the new annotation data is added to the training set; at the same time, the annotated documents are reviewed and updated; the accuracy of topic extraction can be continuously improved, so that the system can better adapt to the changing document content and user needs; for example, in the field of education, with the continuous updating of teaching methods and course content, dynamic updating of annotation data can enable the system to always accurately extract the latest teaching topics;

[0094] Example description:

[0095] Scenario setting of intelligent data information quick check and question-answering system in the field of education:

[0096] Suppose there is an intelligent data information rapid checking and question-answering system for educators and students, covering all kinds of textbooks, teaching materials, test papers, educational research papers and other documents from elementary school to high school.

[0097] The initial state of the document import module:

[0098] Format parsing and subject extraction: The system can initially receive documents in PDF, DOC, TXT, PPT and other formats. For example, when importing a PDF file of a high school mathematics textbook, the system uses the corresponding libraries and tools to parse and extract the text content. Then, it uses machine learning algorithms to preliminarily analyze the document content and extract some subject keywords, such as "high school mathematics", "function", "geometry", etc.

[0099] Establish a mechanism to continuously update annotation data:

[0100] Determine the update cycle: set the manual annotation of newly imported documents every month. For example, in one month, the system may receive some new teaching research papers and the latest simulation test papers.

[0101] Manual annotation of new documents: A professional team of education experts annotates new documents. For example, for a research paper on a new mathematics teaching method, experts annotate the subject keywords such as "high school mathematics", "innovative teaching methods", and "cultivation of mathematical thinking" based on the content of the paper. These annotated documents are added to the training set to update the model of the machine learning algorithm.

[0102] Review and update of annotated documents: Annotated documents are reviewed quarterly. For example, as time goes by, some teaching methods may have been updated or no longer applicable, and experts will re-examine the documents previously annotated as "traditional teaching methods." If it is found that some of these documents actually introduce a modified traditional method that is closer to what is now called a "blended teaching method," the annotations of these documents will be updated, and their subject keywords will be adjusted to "high school mathematics," "blended teaching methods," and so on.

[0103] Actual effect display:

[0104] Adapt to changes in teaching methods and course content: With the continuous development of the education field, teaching methods and course content are also constantly updated. For example, in primary school mathematics teaching, elements of programming education are introduced. By dynamically updating the annotation data, the system can timely and accurately extract documents related to the new teaching topic. When the user asks "How to introduce programming education in primary school mathematics teaching?", the system can quickly find the latest teaching materials and research papers to provide users with accurate answers.

[0105] Improve topic extraction accuracy

[0106] As new annotated data is continuously added to the training set, the accuracy of machine learning algorithms continues to improve. For example, the system may not accurately extract documents on some vague topics at first, but as the annotated data continues to increase, the algorithm can better understand the content of the document and accurately extract the topic. For example, an experimental report on interdisciplinary teaching may initially only extract relatively broad topic keywords such as "disciplinary integration", but after multiple updates, it can accurately extract more specific topic keywords such as "interdisciplinary teaching experiment in elementary school science and mathematics".

[0107] In summary, by establishing a mechanism for continuously updating annotated data, this intelligent data information rapid inspection and question-answering system can better adapt to the ever-changing document content and user needs in the field of education, and provide more accurate and useful information services for educators and students.

[0108] Furthermore, in the result output module, the system also provides multi-angle information display to ensure the comprehensiveness and accuracy of the answers.

[0109] Furthermore, the document import module also includes:

[0110] Preview and Edit Unit: Provides a document preview function before importing, allowing users to view the document content first to ensure that the correct version is imported; allows users to perform simple edits on the document, such as cropping, rotating, or adding annotations, to meet specific import requirements;

[0111] Automatic format detection and conversion unit: The system automatically detects the document format and automatically converts it to a unified format when necessary to reduce the user's manual operations; it provides detailed setting options for format conversion, allowing users to customize conversion parameters to meet specific format requirements;

[0112] Error handling and logging unit: During the document import process, the system handles various error situations, including unsupported file format and file corruption, and gives clear error prompts; records detailed import logs, including import time, imported document list, and errors that occurred, to facilitate subsequent problem troubleshooting and tracking;

[0113] Permission management and security unit: Introduce permission management mechanism so that only authorized users can perform document import operations; perform security checks on imported documents to prevent the introduction of malicious code or viruses.

[0114] Furthermore, the vectorized analysis module also includes:

[0115] Spelling Correction Unit: For misspelled words, use the spelling correction unit tool to correct them to avoid semantic deviations caused by spelling errors;

[0116] Synonym replacement unit: In order to enrich the semantic information of the text, the synonym replacement unit technology is used to replace some words in the text with their synonyms or near synonyms;

[0117] Sentiment analysis unit: In the preprocessing stage, a sentiment analysis unit module is added to extract the sentiment tendency in the text and provide the features of the sentiment dimension for subsequent tasks;

[0118] Topic model unit: Use topic model units (such as LDA) to extract the topic distribution of documents as another important feature of the document;

[0119] Named Entity Recognition Unit: Identifies named entities in text, including names of people, places, and institutions. These entities often carry important information and help improve the effect of vector analysis.

[0120] Context feature unit: Consider using context information to enhance the effect of feature extraction, including using sliding window technology to extract context feature units of words;

[0121] Hybrid embedding unit: combines different types of embedding techniques, including word embedding and document embedding, to build richer vector representations that capture different aspects of text;

[0122] Dynamic embedding unit: Use a model that can dynamically generate word vectors based on context, including ELMo and BERT, to better capture the dynamic semantic information of the text;

[0123] Dimension optimization unit: optimizes the dimension of the vector and selects the appropriate dimension size to balance the amount of information represented by the vector and the computational complexity;

[0124] Semantic similarity measurement unit: In addition to cosine similarity, explore other semantic similarity measurement unit methods, including Jaccard similarity and Euclidean distance, to more accurately evaluate the similarity between documents;

[0125] Clustering and classification unit: Use unsupervised learning methods, including K-means, to cluster vectors, and use supervised learning methods, including SVM and neural networks, to classify and further explore the structure and pattern of text data;

[0126] Visualization and Explanation Unit: Through dimensionality reduction techniques including t-SNE and PCA, high-dimensional vector space is mapped to low-dimensional space for visualization, helping users to intuitively understand the relationship and distribution between documents. At the same time, by providing vector explanations, including keywords and topic tags, users' understanding of vector representation is enhanced;

[0127] Modular design unit: The vector analysis module is designed as a pluggable component, and users can select and combine it according to the needs of specific tasks.

[0128] Furthermore, the dimension optimization unit includes:

[0129] Adaptive dimensionality adjustment subunit: Design an adaptive dimensionality adjustment method to dynamically adjust the dimension size of the vector according to the length, complexity or task requirements of the input text. For example, for shorter texts or simple tasks, lower dimensions can be used to reduce computational costs; for longer texts or complex tasks, the dimensions are increased to capture more information; and introduce an attention mechanism that dynamically adjusts the dimension of the vector according to different parts of the text. For key sentences or paragraphs in the text, more dimensions can be allocated to capture rich information, while for less important parts, fewer dimensions are used; at the same time, introduce dimensionality growth and reduction, and design a model architecture that can dynamically increase or decrease dimensions. At the beginning, lower dimensions can be used to quickly capture basic information, and then the dimensions can be gradually increased as needed to capture more complex features. At the same time, if some dimensions are proven to be unnecessary in subsequent processing, a mechanism can be designed to safely reduce the dimensions;

[0130] Task-based dimension learning subunit: learn the optimal dimension size through the training process; this can be achieved by adding a learnable dimension parameter to the model, which is optimized together with other parameters during the training process; in this way, the model can automatically learn the dimension size that best suits the current task; and condition the dimension parameter: associate the dimension parameter with the task description or text features so that it can be adjusted according to different tasks or text content. This can be achieved by using conditional neural networks or hypernetworks, in which the dimension parameter is part of the network output; at the same time, evaluate the complexity of the task: design a task complexity evaluation module that can analyze the requirements and difficulty of the task and dynamically adjust the dimension of the vector accordingly. For example, for tasks that require deep semantic understanding, the dimension can be increased to capture richer contextual information;

[0131] Dimension sharing and selection of subunits: Design a multi-scale or hierarchical vector representation method in which different dimensions or subspaces represent different semantic information; when using, choose to use all or part of the dimensions according to task requirements to achieve a balance between flexibility and efficiency; in addition to selecting dimensions according to task requirements, design a dimension selection method based on text content, including selecting a specific subset of dimensions based on the topic or sentiment of the text; specifically: first, preprocess the text data, including word segmentation, stop word removal, stemming or word form restoration, etc., to obtain standard vocabulary units to extract text features, which can be achieved in a variety of ways, such as bag-of-words model, TF-IDF, n-gram, etc. These features will serve as the basis for subsequent construction of vector representation, and then construct a multi-scale or hierarchical vector space: The first layer: the basic semantic layer, which contains the most basic semantic information, usually using traditional word embedding methods (such as Word2Vec, G l oVe) generates a vector of fixed dimension for each vocabulary unit. The second layer: contextual semantic layer, using more complex models (such as BERT, GPT and other pre-trained language models) to capture the meaning of words in context and generate higher-level vector representations. The third layer: high-level semantic layer. At this layer, the topic distribution or emotional tendency of the text can be extracted by performing topic modeling (such as LDA) or sentiment analysis units on the text to form a more advanced semantic vector. Finally, dimension selection and task adaptation are performed: according to the specific needs of the task, all or part of the dimensions are selected. For example, for sentiment analysis unit tasks, more attention may be paid to the emotional tendency dimension; for topic classification tasks, more reliance may be placed on the topic distribution dimension. By analyzing the specific content of the text (such as keywords, sentence structure, etc.), the subset of dimensions most relevant to the text topic or emotion is determined. This can be achieved by training an auxiliary model that can predict the best combination of dimensions based on the text content.

[0132] Examples

[0133] Suppose there is a review dataset of an e-commerce platform, and the task is to perform sentiment analysis and topic classification on the reviews;

[0134] Preprocessing: perform preprocessing steps such as word segmentation and stop word removal on the comments;

[0135] Feature extraction: extracting features such as word frequency, TF-I DF, etc. of comments;

[0136] Construct the vector space:

[0137] Basic semantic layer: Use Word2Vec to generate a 50-dimensional vector for each word;

[0138] Contextual semantic layer: Use the BERT model to generate a 768-dimensional context vector for each comment;

[0139] High-level semantic layer: Topic modeling is performed through LDA to obtain the distribution of each comment on K topics; at the same time, sentiment analysis unit tools are used to extract sentiment tendency features;

[0140] Dimension selection:

[0141] For sentiment analysis unit tasks, we can mainly rely on certain dimensions and sentiment tendency features in the context semantic layer;

[0142] For topic classification tasks, we may pay more attention to the topic distribution features generated by LDA;

[0143] Dimension selection based on text content:

[0144] Train an auxiliary model that analyzes keywords and sentence structures in reviews and predicts the most effective combination of dimensions for sentiment analysis units or topic classification tasks;

[0145] Through this method, a multi-scale or hierarchical vector representation method is constructed, which can not only capture multiple levels of text information, but also flexibly choose which dimensions to use according to task requirements, thus achieving a balance between flexibility and efficiency.

[0146] Sparse subunit: Using sparse subunits, including L1 regularization and one of the feature selection methods, some dimensions of the vector are set to zero to reduce computational complexity and storage space. This method can reduce the dimension of the vector while maintaining a certain amount of information;

[0147] Knowledge distillation subunit: Using the knowledge distillation subunit technology, knowledge is extracted from a high-dimensional pre-trained model and transferred to a low-dimensional model, which can take advantage of the advantages of the high-dimensional model while reducing computational complexity and storage requirements.

[0148] Furthermore, the NLP processing module introduces an opinion mining mechanism in the process of identifying and understanding the sentiment tendencies and opinions in the text. In the basic implementation steps of the opinion mining mechanism, the sentiment trend analysis based on time series, multi-dimensional fine-grained opinion mining, and social network analysis and sentiment propagation model are introduced; the sentiment trend analysis based on time series includes the introduction of time series model and sentiment fluctuation measurement, the multi-dimensional fine-grained opinion mining includes the construction of aspect dictionary and opinion association based on knowledge graph, and the social network analysis and sentiment propagation model includes user influence evaluation and sentiment propagation modeling;

[0149] Introducing time series models: In real-time opinion monitoring and analysis, time series analysis methods are used to model the changing trends of opinions; through time series prediction algorithms, changes in sentiment tendencies over a period of time in the future can be predicted, providing early warnings or strategic recommendations for decision makers;

[0150] Sentiment Fluctuation Metrics: Define metrics for sentiment fluctuations, such as the standard deviation or rate of change of sentiment polarity, to quantify how drastic the change in opinion is; useful for identifying the impact of breaking events or hot topics on public sentiment;

[0151] Constructing aspect dictionaries: To mine opinions in texts in a more fine-grained manner, you can construct aspect dictionaries for specific fields, such as product features, service quality, etc. By matching aspect words and opinion words in the text, you can extract specific opinions for each aspect.

[0152] Opinion association based on knowledge graph: Using knowledge graph technology, different opinions are associated and integrated to form a comprehensive evaluation of the overall event or product; it helps to reveal the internal connection and mutual influence between opinions;

[0153] User influence assessment: Combined with social network analysis, evaluate the influence of users in the process of opinion dissemination; identify opinion leaders and key disseminators by calculating indicators such as user degree centrality and betweenness centrality;

[0154] Emotional communication modeling: Establish an emotional communication model to simulate the diffusion process of opinions in social networks; reveal the diffusion mechanism and influencing factors of emotional tendencies by analyzing the transmission path, transmission speed and transmission range;

[0155] Examples:

[0156] Suppose you are monitoring public sentiment on social media about a newly released smartphone; to implement the above innovative improvements, you could do the following:

[0157] Sentiment trend analysis based on time series:

[0158] We collected social media comment data within a week after the phone was released, and used time series analysis to model sentiment. We plotted a line graph of sentiment polarity over time and found that the sentiment polarity was relatively positive in the first two days after the release, but then fluctuated and declined.

[0159] Further analysis of the sentiment fluctuation metric revealed that a large number of negative comments about the battery life of the phone appeared in the comments on a certain day, causing the overall sentiment polarity to drop; this provides an early warning of potential problems;

[0160] Multi-dimensional fine-grained opinion mining:

[0161] We built a dictionary of aspects for smartphones, including screen, camera, performance, battery life, etc. We extracted specific opinions on each aspect by matching aspect words and opinion words in the reviews.

[0162] By using knowledge graph technology, we can associate and integrate different viewpoints to form an overall evaluation graph of mobile phones. Through visualization, we can intuitively see which aspects have received positive or negative reviews from users.

[0163] Social network analysis and sentiment propagation model:

[0164] Using social network analysis methods, the influence index of comment users in social networks was calculated. By identifying opinion leaders and key communicators, it was found that some users with greater influence played a key role in the spread of emotional tendencies.

[0165] Based on the sentiment propagation model, the diffusion process of opinions in social networks is simulated, and the propagation path and speed are analyzed, which helps to understand how sentiment tendencies spread and evolve in social networks.

[0166] Through the above innovative improvements and practical applications, we can more accurately monitor and analyze real-time changes in opinions on social media, mine fine-grained opinion information, and reveal the propagation mechanism and influencing factors of emotional tendencies, which is of great significance for corporate decision-making, product improvement and public opinion response.

[0167] Furthermore, in the question-answer matching module, pre-trained language models are introduced and combined with knowledge graphs, that is, pre-trained language models such as BERT and GPT are used to enhance the performance of question-answer matching. These models are trained with a large amount of text data and can better understand the semantics and context of the question (or entity linking technology, that is, associating the question with the entities and relationships in the knowledge base to provide more accurate and comprehensive answers); and personalization and context awareness are introduced to add personalized elements to question-answer matching based on the user's personal preferences and historical behavior data, and use the conversation context information to understand the coherence and intention of the user's questions, improve the coherence and accuracy of the answers, and improve the module's ability to handle complex problems by introducing a technology in logical reasoning or causal reasoning; by combining pre-trained language models, knowledge graphs, and personalization and context awareness technologies, the question-answer matching module can understand the user's questions more accurately and provide more targeted and accurate answers; at the same time, the introduction of technologies such as logical reasoning and causal reasoning can further improve the system's ability to handle complex problems and enhance user experience.

[0168] The imported documents will first be vectorized, which will convert the text that humans can understand into vectors that machines can understand, and classify and organize the vector features according to vocabulary, grammar, and context. When a user asks a question, the question will be vectorized in the same way, and then the vector in the library will be compared with the vector of the question to match the degree, and the x most matching results will be returned. Finally, the matching results will be summarized and organized into paragraphs and converted into human language according to LLM and returned.

Claims

1. An intelligent data information rapid inspection and question-answering system, characterized in that: It includes document import module, vectorization analysis module, NLP processing module, question-answer matching module and result output module; Document import module: used to receive and import documents including but not limited to PDF, DOC, TXT, and PPT. It uses corresponding libraries and tools to parse documents of different formats. When processing test questions and test paper format files, it identifies and classifies the files by identifying the title format and question number format. It also uses machine learning algorithms to conduct preliminary analysis of the document content and extract subject keywords for subsequent retrieval and processing. Vectorization analysis module: Use text vectorization technology to pre-process the imported document content and then input it into the Transformer model to obtain the vector representation of each word or sentence. Use one of the TF-IDF algorithm and TextRank algorithm to extract keywords, sentences, and paragraphs, and map them to points in the high-dimensional vector space to identify and summarize the key information and structure of the document. NLP processing module: Using natural language processing technology, the vectorized document content is subjected to part-of-speech tagging, named entity recognition, and syntactic analysis to obtain vocabulary, grammar, and context information, identify the document theme, key information, and structure, use the topic model for topic identification, and save the document content according to the preset classification system, establish a label system, and classify it into corresponding categories for subsequent retrieval and matching; Question-answer matching module: pre-process the text, convert the question words into vectors using the pre-trained Word2Vec model, and then get the question vector representation by averaging and summing. TF-IDF is combined with the bag-of-words model to calculate the TF-IDF value of each word in the question to construct the bag-of-words vector. Cosine similarity and Euclidean distance are used in the vector space. Each document in the document library is also vectorized in the same way as the question, and the document vector is calculated and stored when the document is imported. The semantic structure of the question is analyzed with the help of dependency syntax analysis in natural language processing technology, and the key entities and attributes of the question are determined. Search for content with the same or similar entities and attributes in the document; use the topic model LatentDirichletAllocation to extract the document topic, and consider the relevance of the question topic and the document topic when matching questions. If the topics are similar, the document is more likely to be the answer to the question; for test questions and solutions, the module associates them based on keywords and key steps; extract keywords from test questions and solution documents respectively through the TF-IDF keyword extraction algorithm, and then use similarity algorithms such as cosine similarity to calculate the similarity between keywords based on the matching degree of keywords, and determine the degree of association between test questions and solutions; Result output module: organizes the matching results into paragraphs, organizes them reasonably according to question type and document structure, extracts relevant sentences and paragraphs to organize answers, converts them into human language using language models and returns them to users, controls the output style and accuracy by adjusting language model parameters, supports test paper generation and formatted output, and is used to customize the output test paper format and content according to user needs.

2. An intelligent data information rapid inspection and question-answering system as claimed in claim 1, characterized in that: The document import module includes a user interface design unit, a document format conversion unit, a batch import unit, and a document storage and management unit.

3. An intelligent data information rapid inspection and question-answering system as claimed in claim 1, characterized in that: In the document import module, a mechanism is established to continuously update the annotation data. With the continuous import of new documents and the use of the system, new documents are manually annotated regularly and the new annotation data is added to the training set. At the same time, the annotated documents are reviewed and updated.

4. An intelligent data information rapid inspection and question-answering system as claimed in claim 1, characterized in that: In the result output module, the system also provides multi-angle information display.

5. An intelligent data information rapid inspection and question-answering system as claimed in claim 2, characterized in that: The document import module also includes a preview and editing unit, an automatic format detection and conversion unit, an error handling and logging unit, and a permission management and security unit.

6. An intelligent data information rapid inspection and question-answering system as claimed in claim 1, characterized in that: The vectorized analysis module also includes a spelling correction unit, a synonym replacement unit, a sentiment analysis unit, a topic model unit, a named entity recognition unit, a context feature unit, a hybrid embedding unit, a dynamic embedding unit, a dimension optimization unit, a semantic similarity measurement unit, a clustering and classification unit, a visualization and explanatory unit, and a modular design unit.

7. An intelligent data information rapid inspection and question-answering system as claimed in claim 6, characterized in that: The dimension optimization unit includes an adaptive dimension adjustment subunit, a task-based dimension learning subunit, a dimension sharing and selection subunit, a sparsification subunit, and a knowledge distillation subunit.

8. An intelligent data information rapid inspection and question-answering system as claimed in claim 1, characterized in that: In the process of identifying and understanding the sentiment tendencies and opinions in the text, the NLP processing module introduces the opinion mining mechanism. In the basic implementation steps of the opinion mining mechanism, time-series-based sentiment trend analysis, multi-dimensional fine-grained opinion mining, and social network analysis and sentiment propagation models are introduced; time-series-based sentiment trend analysis includes the introduction of time series models and sentiment fluctuation measurements, multi-dimensional fine-grained opinion mining includes the construction of aspect dictionaries and opinion associations based on knowledge graphs, and social network analysis and sentiment propagation models include user influence evaluation and sentiment propagation modeling.

9. An intelligent data information rapid inspection and question-answering system as claimed in claim 8, characterized in that: In the question-answer matching module, a pre-trained language model is introduced and combined with a knowledge graph; And introduce personalization and context awareness, add personalized elements to question-answer matching based on users' personal preferences and historical behavior data, and introduce a technique in logical reasoning or causal reasoning.

Images