An education intelligent question and answer method and system based on multi-modal knowledge resolution

Abstract

This application discloses an educational intelligent question-answering method and system based on multimodal knowledge parsing, belonging to the field of computer technology. The method first performs structured parsing of textbook materials and generates semantic descriptions for each first-type element based on the parsing results. This transforms the multimodal textbook content information into indexable knowledge representations. Combined with the textbook layout and location information, a unified list of content elements is generated to balance preserving the original textbook structure with utilizing multimodal knowledge, thus ensuring a stable correlation between the generated results and the original textbook text. Then, the unified list of content elements is structured into blocks to ensure the referability and traceability of the knowledge blocks. Based on these knowledge blocks, a multimodal textbook knowledge base is constructed to support subsequent retrieval and the generation of answers with references. This effectively solves the problems of unclear answer basis, difficulty in traceability, and poor quality in educational intelligent question-answering systems.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to an educational intelligent question-answering method and system based on multimodal knowledge parsing. Background Technology

[0002] With the advancement of large language models and intelligent agent technology, the application of artificial intelligence in education has gradually expanded from traditional knowledge question answering to diversified scenarios such as learning feedback, content generation, personalized tutoring, and learning process support.

[0003] However, current educational intelligent question-answering systems typically rely directly on large language models to generate content, resulting in a lack of stable correlation between the generated results and the original textbook text, leading to unclear and untraceable basis for the answers. At the same time, current systems usually only process text content (such as the text content of the textbook body) and lack effective processing methods for multimodal teaching resources such as images and tables, making it difficult to fully utilize textbook knowledge and resulting in poor answer quality. Summary of the Invention

[0004] The purpose of this application is to provide an educational intelligent question-answering method and system based on multimodal knowledge parsing, which can solve the problems of unclear answer basis, difficulty in traceability and poor quality in related technologies' educational intelligent question-answering systems.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide an educational intelligent question-answering method based on multimodal knowledge parsing, the method comprising: In response to receiving textbook materials uploaded by a user, the textbook materials are subjected to structured parsing to obtain parsing results, which include: multimodal textbook content information, and textbook layout and location information; Based on the textbook content information and context information of each first type element in the parsing results, a semantic description of each first type element is generated, wherein different types of elements correspond to different modalities, and each first type element is a non-text type element; The parsing results and the semantic descriptions of each of the first type elements are organized into a unified content element list; The unified content element list is structured and divided into multiple knowledge blocks; Based on the aforementioned knowledge blocks, a multimodal textbook knowledge base is constructed; In response to detecting the query text to be retrieved associated with the user's question, the system retrieves the various knowledge blocks associated with the query text based on the multimodal textbook knowledge base. After retrieving all the query texts associated with the user's question, an answer containing textbook citation information is generated based on the knowledge blocks associated with all the retrieved query texts.

[0006] Secondly, embodiments of this application provide an educational intelligent question-answering system based on multimodal knowledge parsing, the system comprising: The multimodal textbook knowledge organization module is used to respond to received textbook materials uploaded by users, perform structured parsing on the textbook materials to obtain parsing results, the parsing results including: multimodal textbook content information, and textbook layout and position information; and generate semantic descriptions for each first-type element based on its respective textbook content information and context information in the parsing results, wherein different types of elements correspond to different modalities, and each first-type element is a non-text element; and organize the parsing results and the semantic descriptions of each first-type element into a unified content element list; and perform structured segmentation of the unified content element list to obtain multiple knowledge blocks; and construct a multimodal textbook knowledge base based on the multiple knowledge blocks; The textbook evidence retrieval module is used to respond to the detection of the query text to be retrieved associated with the user's question, and to retrieve the various knowledge blocks associated with the query text based on the multimodal textbook knowledge base; The answer generation module is used to generate an answer containing textbook citation information based on the knowledge blocks associated with all the retrieved query texts after completing the retrieval of all the query texts associated with the user's question.

[0007] Thirdly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the educational intelligent question-answering method based on multimodal knowledge parsing as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the educational intelligent question-answering method based on multimodal knowledge parsing as described in the first aspect.

[0009] This application provides an effective processing method for multimodal teaching resources. First, the textbook materials are structurally parsed, and semantic descriptions are generated for each first-type element based on the parsing results to make the relevant modal textbook content information indexable. This transforms the multimodal textbook content information in the textbook materials into indexable knowledge representations. Combined with the textbook layout and location information, a unified list of content elements is generated to balance the preservation of the original textbook structure and the utilization of multimodal knowledge, thereby ensuring a stable correlation between the subsequent generated results and the original textbook text. Then, the unified list of content elements is structurally segmented to ensure the referability and traceability of the knowledge blocks. Based on the knowledge blocks, a multimodal textbook knowledge base is constructed to support subsequent retrieval and the generation of answers with references. This can effectively solve the problems of unclear answer basis, difficulty in traceability, and poor quality in educational intelligent question-answering systems. Attached Figure Description

[0010] Figure 1 A flowchart illustrating the implementation of an educational intelligent question-answering method based on multimodal knowledge parsing, provided in an embodiment of this application; Figure 2 A hybrid retrieval flowchart is provided for an embodiment of this application; Figure 3 A flowchart of PDF file parsing and multimodal preprocessing is provided for embodiments of this application; Figure 4 A schematic diagram illustrating the construction of a title tree as provided in an embodiment of this application; Figure 5 A Deep Research workflow diagram provided for embodiments of this application; Figure 6 A flowchart illustrating the implementation of an interactive teaching closed loop is provided in this application embodiment; Figure 7 An overall architecture diagram of an educational intelligent agent system provided in this application embodiment; Figure 8 A schematic diagram illustrating an end-to-end test result comparison of hybrid retrieval and report generation, provided as an embodiment of this application; Figure 9 A schematic diagram illustrating the structured learning report generation effect (test group A) provided in an embodiment of this application; Figure 10 A schematic diagram of an interactive teaching closed-loop test result provided in an embodiment of this application; Figure 11 A schematic diagram of a source citation card provided for an embodiment of this application; Figure 12 This is a schematic diagram of a research history session list provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0013] Current intelligent question-answering systems in education typically rely directly on large language models to generate content, failing to integrate textbook content, heading levels, and page layout information into the knowledge organization, retrieval, and generation processes. This results in a lack of stable correlation between the generated results and the original textbook text, leading to unclear and difficult-to-trace the basis of the answers. Furthermore, current systems usually only process text content; for example, they typically only extract, segment, and vectorize the text from the main body of the textbook, lacking effective methods for unified parsing and knowledge organization of multimodal teaching resources (such as images and tables) within the textbook, thus preventing the effective retrieval of key information.

[0014] Specifically, current Retrieval-augmented Generation (RAG) educational question-answering systems primarily rely on textbook text knowledge bases. These systems typically include modules for document uploading, text extraction, text segmentation, vectorization, vector database storage, question retrieval, and large language model generation. The specific workflow is as follows: After a user uploads a Portable Document Format (PDF) file, course handouts, or other teaching materials, the system first extracts the text content from the document and segments it into several text blocks based on fixed length, natural paragraphs, or sliding windows. Then, it calls a text embedding model to convert these text blocks into vector representations and stores the text content and its corresponding vectors in a vector database. When a question is entered, the system converts the user's question into a query vector and retrieves several semantically similar text blocks from the vector database. Finally, it concatenates the retrieved text blocks with the user's question to form prompt words, which are then input into a large language model to generate a user-oriented answer.

[0015] While the aforementioned systems can incorporate textbook knowledge to some extent, their core reliance on text extraction, segmentation, and vector retrieval makes it difficult to fully process multimodal information in textbook materials, such as images, tables, heading levels, page numbers, and page layouts. This results in incomplete utilization of textbook knowledge, which in turn affects the quality of the answers. Furthermore, the method of generating answers directly based solely on text blocks leads to a lack of stable correlation between the generated results and the original textbook text, making the basis of the answers unclear and difficult to trace.

[0016] To address the aforementioned issues, this application provides an educational intelligent question-answering method and system based on multimodal knowledge parsing. This method can uniformly transform different types of elements in teaching materials into knowledge units that are searchable, citationable, and traceable, thereby constructing a multimodal teaching material knowledge base and integrating teaching material content into the process of knowledge organization, retrieval, and generation.

[0017] The following description, in conjunction with the accompanying drawings, details an educational intelligent question-answering method and system based on multimodal knowledge parsing provided in this application, through specific embodiments and application scenarios.

[0018] See Figure 1 The diagram shown is an implementation flowchart of an educational intelligent question-answering method based on multimodal knowledge parsing provided in this application embodiment. The method may include the following steps: Step S101: In response to receiving the textbook materials uploaded by the user, perform structured parsing on the textbook materials to obtain the parsing results. The parsing results include: multimodal textbook content information, and textbook layout and location information.

[0019] In this embodiment, considering that teaching materials containing multimodal information (such as textbook PDF files) are usually page layout descriptions rather than text structures with a clear semantic hierarchy, they need to be converted into structured content first for subsequent knowledge organization and retrieval.

[0020] Specifically, after receiving textbook PDF files and other teaching materials uploaded by users, the system first performs structured parsing on the text, images, tables, formulas, page numbers, heading levels, and local layout positions in the teaching materials to obtain parsing results containing structured content. The multimodal teaching material content information can include text information (such as the main text, titles, and chapter content), image information (such as image elements, screenshots, titles, and contextual information in the teaching materials), table information (such as table elements, screenshots, titles, content, and contextual text), and formula information (such as various formulas in the teaching materials). The teaching material layout and position information can include page numbers, paths, and local position information in the teaching materials.

[0021] As one possible implementation, the teaching materials may include PDF files, Word documents, PPT presentations, scanned handouts, web-based teaching materials, e-books, or other teaching resources.

[0022] As one possible implementation, before performing structured parsing on the teaching materials, file verification or document archiving operations can be performed on the teaching materials.

[0023] Taking textbook materials including PDF files as an example, after receiving the textbook PDF file uploaded by the user, the system first performs file verification and document archiving operations, and then calls the PDF structured parsing module to parse the original textbook PDF file; wherein, the file verification includes at least one of the following: File type validation: Determines whether the file uploaded by the user is a PDF; File size verification: Checks whether the size of the file uploaded by the user exceeds the system's allowed processing range; File integrity verification: Checks whether user-uploaded files are empty, corrupted, or cannot be read normally by the PDF parsing module; File ownership verification: Check whether the user-uploaded file is bound to a valid knowledge base or course resource directory; File duplication check: It can determine whether there are duplicate uploads based on file name, file size, file hash value or existing document records in the knowledge base, thus avoiding duplicate parsing and duplicate entry into the database; File save path verification: After saving the original PDF file, check whether the file path was generated successfully and whether the file can be accessed by subsequent parsing tasks, thus providing valid input for the background asynchronous parsing task.

[0024] Step S102: Based on the textbook content information and context information of each first type element in the parsing result, generate the semantic description of each first type element. Different types of elements correspond to different modalities, and each first type element is a non-text type element.

[0025] In this embodiment, semantic descriptions are generated for non-textual elements (i.e., first-type elements) such as image and table elements in teaching materials, in order to transform non-textual content such as images and tables into searchable semantic knowledge representations. Thus, by first semantically enhancing the first-type elements (e.g., converting images and tables into semantic descriptive text), and then incorporating them into subsequent knowledge organization, segmentation, database entry, and retrieval processes, the problem of non-textual raw objects such as images and tables in teaching materials being unindexable is solved.

[0026] Step S103: Organize the parsing results and the semantic descriptions of each of the first type elements into a unified content element list.

[0027] In this embodiment, the parsing results and the semantic descriptions of each of the first-type elements are organized into a unified content element list, thereby providing basic data for subsequent knowledge segmentation and storage. It is understood that in this unified content element list, multimodal content such as images and tables, which were originally difficult to retrieve directly, are transformed into indexable knowledge representations (i.e., semantic descriptions), while retaining their original page numbers, regional locations, and contextual relationships, thus balancing the preservation of the original textbook structure with the utilization of multimodal knowledge.

[0028] Step S104: The unified content element list is structurally segmented to obtain multiple knowledge blocks.

[0029] In this embodiment, unlike the traditional method of directly dividing the entire textbook into sliding window segments according to a fixed length, this application adopts a structured segmentation method. For example, the unified content element list can be segmented according to the title tree and chapter path so as to cut it into independent knowledge units (i.e. knowledge blocks) according to the inherent logical hierarchy or format marks of the data, thereby ensuring the referenceability and traceability of the knowledge units.

[0030] As one possible implementation, the structured segmentation method may include single segmentation methods or combinations of multiple segmentation methods such as segmentation based on title tree and chapter path, paragraph segmentation, semantic segmentation, and knowledge point segmentation.

[0031] Step S105: Construct a multimodal textbook knowledge base based on the multiple knowledge blocks.

[0032] In this embodiment, a multimodal textbook knowledge base can be constructed based on full-text indexing or vector indexing for the multiple knowledge blocks to support full-text retrieval or vector retrieval of knowledge blocks. Specifically, full-text indexing is used to match knowledge blocks at the keyword level to prioritize the retrieval of knowledge blocks containing specific target terms, title words, fixed expressions, textbook concept names, formula names, chapter titles, or proprietary technical terms; vector indexing is used to match knowledge blocks at the vector level to prioritize the retrieval of semantically similar knowledge blocks.

[0033] As one possible implementation, each knowledge block contains the following information: block number, document number, block type, knowledge block text, title path, start and end page numbers, original element number, page area coordinates, and contextual information (such as surrounding text).

[0034] Step S106: In response to detecting the query text to be retrieved associated with the user's question, retrieve the various knowledge blocks associated with the query text based on the multimodal textbook knowledge base.

[0035] In this embodiment, the query text can be generated based on the user's input question (such as a learning topic or task entered by the user), or it can be generated and input by other systems or intelligent agents based on the user's question. Each user question can be associated with one or more query texts (such as the query texts corresponding to multiple sub-questions).

[0036] For the query text to be retrieved associated with the user's question, based on the multimodal textbook knowledge base, knowledge blocks are retrieved by means of full-text search or vector search, and the top-ranked knowledge blocks (such as the top k knowledge blocks with the highest semantic similarity to the query text) are determined as the knowledge blocks associated with the query text.

[0037] Step S107: After completing the retrieval of all query texts associated with the user's question, generate an answer carrying textbook citation information based on the knowledge blocks associated with all the retrieved query texts.

[0038] In this embodiment, a large language model can be used to generate an answer to the user's question by using all the knowledge blocks associated with the user's question as teaching material evidence. During the answer generation process, teaching material citation information is generated based on page numbers, title paths, and original area information in the relevant knowledge blocks to ensure that the generated answer can accurately trace back to the original text of the teaching material. This enhances the credibility and explainability of the teaching content from a mechanism perspective.

[0039] As can be seen from the above technical solution, this application provides an effective processing method for multimodal teaching resources. First, the textbook materials are structurally parsed, and semantic descriptions are generated for each first-type element based on the parsing results to make the relevant modal textbook content information indexable. This transforms the multimodal textbook content information in the textbook materials into indexable knowledge expressions. Combined with the textbook layout and location information, a unified list of content elements is generated to balance the preservation of the original textbook structure and the utilization of multimodal knowledge, thereby ensuring a stable correlation between the subsequent generated results and the original textbook text. Then, the unified list of content elements is structurally segmented to ensure the referability and traceability of knowledge blocks. Based on the knowledge blocks, a multimodal textbook knowledge base is constructed to support subsequent retrieval and the generation of answers with references. This can effectively solve the problems of unclear answer basis, difficulty in traceability, and poor quality in educational intelligent question-answering systems.

[0040] In some embodiments, the multimodal textbook knowledge base includes a structured database and a vector database; the step of constructing the multimodal textbook knowledge base based on the plurality of knowledge blocks includes: Each knowledge block and its source metadata are written into the structured database, and each knowledge block is written into the vector database; The step of retrieving various knowledge blocks associated with the query text based on the multimodal textbook knowledge base includes: The query text is processed into a query vector using an embedding model, and semantically similar knowledge blocks are retrieved from the vector database based on the query vector to obtain vector retrieval results; Based on the structured database, keyword retrieval is performed on the query text to recall the query terms associated with the query text, as well as knowledge blocks that match at least one of chapter titles, formula titles, and proper nouns, to obtain full-text retrieval results; Based on the vector retrieval results and the full-text retrieval results, the knowledge blocks associated with the query text are determined.

[0041] In this embodiment, considering that current educational systems based on retrieval enhancement often rely on single-path retrieval (such as single vector retrieval), when faced with situations where textbook knowledge is scattered, expressions are diverse, content has cross-chapter connections, and chart-related content is difficult to vectorize, problems such as incomplete textbook evidence recall or unstable recall results are likely to occur. Therefore, see [link to relevant documentation]. Figure 2 The flowchart shown illustrates the hybrid retrieval process. This application employs a fusion of full-text retrieval and vector retrieval to achieve the retrieval of textbook evidence.

[0042] Specifically, after completing the structured segmentation, a structured storage and retrieval index is established. The knowledge block content, along with knowledge block source metadata such as page ranges, title paths, and original region information, is written into a structured database (which can be built based on relational databases, etc.) to support full-text retrieval. Simultaneously, knowledge block vectors are written into a vector database to support vector retrieval. Thus, a multimodal textbook knowledge base that simultaneously supports vector retrieval, full-text retrieval, and original text source retrieval can be constructed.

[0043] Based on this, for the query text to be retrieved, the query text is first input into the embedding model to generate a query vector, and then semantically similar knowledge blocks are retrieved from the vector database as vector retrieval results. Simultaneously, keyword retrieval is performed based on the full-text index in the structured database to recall knowledge blocks that precisely match the query terms, textbook terminology, chapter titles, or key concepts as full-text retrieval results. It is understandable that both of these retrieval methods are constrained by the scope of the knowledge base or the scope of the teaching materials, effectively preventing non-target textbook content from entering the candidate results.

[0044] After completing two independent searches, the obtained vector search results and full-text search results can be uniformly merged and sorted to obtain the various knowledge blocks associated with the query text. For example, after the user inputs a learning topic or the agent generates a subquery, the system will perform a keyword search based on the full-text index of the chunks table in PostgreSQL (corresponding to a structured database) within the specified document primary key (document_id) range to recall knowledge blocks that match the query terms, chapter titles, formula titles, or proper nouns. Subsequently, the system merges and sorts the full-text search results with the vector search results based on Milvus (corresponding to a vector database) to form the final set of textbook evidence.

[0045] It should be noted that the above-described hybrid retrieval method combines the capabilities of semantically similar content retrieval and precise keyword matching. Vector retrieval is used to discover content that is semantically related but expressed differently; full-text retrieval is used to accurately capture specialized terminology, formula names, chapter titles, and fixed keywords given in the textbook. These two retrieval paths each have their own focus and complement each other, thereby comprehensively improving the coverage and stability of the textbook evidence retrieval process.

[0046] As one possible implementation, the knowledge block source metadata, user information, knowledge base information, document information, and knowledge block information associated with each knowledge block can be written into the structured database together to support the accurate location of knowledge blocks and the retrospective review of the original textbook text.

[0047] In some embodiments, determining the knowledge blocks associated with the query text based on the vector retrieval results and the full-text retrieval results includes: For each retrieved knowledge block, a comprehensive ranking score is determined based on the ranking of the knowledge block in the vector retrieval results and the full-text retrieval results, respectively. Based on the comprehensive ranking score, all retrieved knowledge blocks are filtered to obtain the knowledge blocks associated with the query text.

[0048] In this embodiment, a reverse sorting fusion method is adopted. The corresponding comprehensive ranking score is calculated based on the ranking of the knowledge block in the two search results. This gives higher priority to knowledge blocks that are recalled by both searches and have a higher ranking, so that they can be selected first in the screening stage.

[0049] In some embodiments, generating semantic descriptions for each of the first-type elements based on their respective textbook content information and context information in the parsing results includes: For each image type element, a semantic description of the image type element is generated based on the image title, screenshot, and surrounding text in the parsing results. For each table-type element, a semantic description of the table-type element is generated based on the table screenshot, table title, table content, and surrounding text in the parsing results. The method further includes: For table-type elements that span multiple pages, before generating semantic descriptions, the information associated with the table-type elements in the parsing results is merged in advance based on the page-spanning relationship between adjacent pages.

[0050] In this embodiment, a multimodal preprocessing method suitable for images and tables is provided. Specifically, for image type elements, the associated image title and screenshot, as well as surrounding text and other contextual information, are extracted, and a multimodal model (such as a multimodal large language model) is called to generate a semantic description (such as image description text) that is easy to retrieve later, so as to retain the knowledge information therein. Optionally, the semantic description can be written back to the corresponding element structure to improve the generation efficiency of the unified content element list.

[0051] For table-type elements, extract their associated table screenshots, table titles, table content, and surrounding text, and use a multimodal model (such as a multimodal large language model) to generate a semantic description (such as table description text) to facilitate subsequent retrieval, thus preserving the knowledge information contained therein. Optionally, this semantic description can be written back to the corresponding element structure to improve the efficiency of generating a unified list of content elements.

[0052] Furthermore, for table-type elements that span multiple pages (i.e., cross-page tables), the relevant content in the parsing results will be merged in advance based on the cross-page relationship between adjacent pages, and then a unified description will be generated for the cross-page table accordingly, thereby reducing the risk of table semantics being fragmented by page boundaries.

[0053] Therefore, by semantically enhancing images and tables, it is possible to preserve the layout structure and element information of the textbook while supplementing the searchable representation of images and tables, so that the knowledge information in images and tables can enter the subsequent knowledge organization and retrieval chain.

[0054] For example, see Figure 3The flowchart shown illustrates the PDF file parsing and multimodal preprocessing process. For a textbook PDF file to be parsed, the pre-packaged DocumentParser module is first called. MinerU is then used to perform structured parsing of the textbook PDF file, generating a content_list, an image directory, and page number information. The content_list stores elements such as text, images, tables, and formulas, and records the page number, path, and local location information for each element.

[0055] After basic parsing, the process doesn't immediately proceed to the segmentation stage. Instead, it first performs multimodal preprocessing on the content_list (corresponding to the parsed results) based on the image directory. For image elements, the image title, screenshot, and contextual information are extracted. A multimodal model is then used to generate a semantic description that facilitates subsequent retrieval, and this description is written back into the corresponding element structure. For table elements, the table screenshot, title, content, and surrounding text are extracted to generate a table description text, preserving the knowledge information within. If a table spans multiple pages, the cross-page relationship between adjacent pages is detected, and related content is merged to generate a unified description, reducing the risk of table semantics being fragmented by page boundaries.

[0056] After the above processing, different types of content such as text, images, tables, and formulas, as well as their page numbers, paths, and local location information, are all uniformly organized in the content_list to obtain the enhanced content_list, which can be used as a unified input for subsequent title tree construction, structured segmentation, and knowledge storage.

[0057] It should be noted that during the PDF structured parsing stage, the generated content_list stores elements such as text, images, tables, and formulas, and records their page numbers, paths, and local location information. During the multimodal preprocessing stage, the semantic description generation utilizes element titles, screenshots, and contextual information simultaneously, thus preserving the contextual relationships. In the knowledge block storage stage, the chunks table further records the content, title path, start and end page numbers, associated image resources, and full-text search fields of the knowledge blocks. Simultaneously, the chunk_source_elements table records the correspondence between knowledge blocks and original content elements, including the original content list index, element type, page number index, region coordinates, image paths, and element order. This allows for the effective preservation of original page numbers, region locations, and contextual relationships.

[0058] In some embodiments, the structured segmentation of the unified content element list to obtain multiple knowledge blocks includes: For the information associated with each first type element in the unified content element list, the information is combined into various multimodal knowledge blocks according to the title tree and chapter boundaries. Each first type element includes image type elements and / or table type elements. For the information associated with each second type element in the unified content element list, the information is merged according to semantic continuity and preset length constraints based on the title tree and chapter boundaries to obtain each text knowledge block. Each second type element includes text type elements and / or formula type elements.

[0059] This embodiment provides a structured segmentation method suitable for first-type elements and second-type elements. Specifically, see [link to documentation]. Figure 4 The flowchart shown illustrates the title tree construction, structured chunking, and knowledge storage process. For first-type elements such as images and tables, relevant information from a unified list of content elements—including their associated semantic descriptions, titles, context windows, and original positions—is combined into various multimodal knowledge chunks (e.g., image chunks, table chunks) based on the title tree and chapter boundaries. For cross-page tables, the corresponding knowledge chunks are generated based on the merged unified description.

[0060] For information associated with second-type elements such as continuous text and formula content, this information can be merged into various text chunks based on semantic continuity and preset length constraints, according to the title tree and chapter boundaries.

[0061] Through the above processing, the originally scattered text, images, tables, and chapter structures in the textbook PDF file can be uniformly organized into searchable knowledge units, thus avoiding the problem that ordinary text RAG systems often ignore chart information.

[0062] In some embodiments, the title tree is constructed through the following steps: Based on the title numbering, font style, page position, and text pattern of the teaching materials, as well as the preset table of contents noise filtering rules, the chapter levels of the teaching materials are identified. The preset table of contents noise filtering rules include: title filtering rules based on at least one of the following: numbering pattern, page number position, table of contents items, cover text, and miscellaneous chart numbers. A title tree is constructed based on the chapter hierarchy of the textbook. Each node in the title tree corresponds to a chapter or section. The node records the title name, level, start and end page numbers, parent-child relationship, and title path of the corresponding chapter or section.

[0063] In this embodiment, as Figure 4As shown, during the title tree construction phase, a preset directory noise filtering rule is introduced to eliminate the interference of directory entries and noisy titles on subsequent index construction.

[0064] In practice, text-level elements can be extracted from the parsing results or the unified content element list as candidate titles. Then, the candidate titles are filtered using preset directory noise filtering rules. Subsequently, based on the filtered candidate titles, the chapter level can be inferred, the content scope of the leaf nodes can be defined, and then a title tree can be constructed.

[0065] In some embodiments, the method further includes: In response to receiving a user question containing a learning topic, the following steps are executed using the agent workflow: Pre-test questions are generated based on the learning topic, and the user's answers are recorded; Task planning is performed on the learning topic to generate query texts corresponding to each sub-query, and evidence reflection and supplementary retrieval are performed on the textbook evidence units associated with each query text; The step of generating an answer carrying textbook citation information based on the knowledge blocks associated with all retrieved query texts includes: The target content of each knowledge block is completed to obtain a textbook evidence unit for the intelligent agent workflow. The target content includes at least one of the following: title path, page number range, block type, text summary and source information. Using the intelligent agent workflow, a personalized learning report and cited references are generated based on the user's answer and all textbook evidence units associated with the learning topic. The cited references are used to record the correspondence between the evidence numbers cited in the personalized learning report and multiple preset dimension information, including knowledge block number, document number, title path, page range, block type, and content summary. Based on the personalized learning report and cited references, generate answers that include textbook citation information.

[0066] In this embodiment, considering that current systems mostly adopt a simple "direct generation after retrieval" mode, lacking mechanisms such as task planning, evidence reflection, and supplementary retrieval, resulting in insufficient logical organization and explanatory power of the generated content; and that current systems typically only support single question-and-answer sessions, failing to link pre-testing, learning analysis, and learning report generation to identify weak knowledge points based on the user's learning foundation and generate personalized learning reports accordingly, this application introduces an intelligent agent workflow to realize pre-testing, task decomposition, evidence retrieval, evidence reflection, and the generation of personalized learning reports with citations.

[0067] For example, see Figure 5 The Deep Research workflow diagram shown illustrates that when a user initiates a learning task for any learning topic, they enter a Deep Research agent workflow comprised of nodes such as Planner, Researcher, Reflection, and Writer (which can be built based on pre-trained models such as large language models, eliminating the need for model training). This workflow completes tasks such as task decomposition, evidence retrieval, evidence reflection, and generation of a learning report with citations. The Planner, Researcher, Reflection, and Writer nodes can be merged, split, or replaced as needed. This addresses the problems of insufficient decomposition of complex learning topics and uninterpretable generation criteria in traditional RAG educational question-answering systems, which directly call large language models to generate the final answer after initial retrieval.

[0068] Before formally entering the learning process, pre-test questions are generated around the current learning topic and students' answers are recorded. Based on this, a learning analysis result is generated, which will be used as personalized input for subsequent learning report generation and multiple rounds of tutoring by AI Tutor.

[0069] The Planner performs task planning for the current learning topic to generate query texts corresponding to each subquery. After the Researcher completes the retrieval of all query texts associated with the user's question using a hybrid retrieval method, it can supplement the title path, page range, block type, text summary, and source information of each knowledge block associated with the user's question based on existing or newly added related knowledge blocks (such as semantically related or adjacent knowledge blocks), forming textbook evidence units for the agent's workflow. Subsequently, Reflection performs evidence reflection on each textbook evidence unit, such as checking whether the textbook evidence unit has sufficient information (e.g., whether it meets the requirement of sufficient knowledge blocks), and supplementing the retrieval of textbook evidence units with insufficient information (e.g., reducing the matching accuracy requirement of knowledge blocks).

[0070] During report generation, the Writer uses prompts and other methods to ensure the generated content uses authentic evidence numbers for citations. After report generation, a citation map is constructed to clearly and completely record the correspondence between citation numbers and multiple preset dimensions of information. This clear and explicit correspondence ensures that every key conclusion presented in the learning report is accurately matched to its corresponding source textbook knowledge block, preventing issues such as ambiguous sources or misaligned correspondences. This provides a solid and complete data foundation for subsequent front-end operations such as viewing cited content and tracing back to the original text. After obtaining the personalized learning report and citations, it can be used directly as an answer incorporating textbook citation information, or combined with other information (such as user questions or follow-up inquiries) to generate answers that also incorporate textbook citation information.

[0071] In this example, by combining hybrid retrieval with a Deep Research workflow, this application avoids the shortcomings of ordinary systems that "directly generate results after retrieval." As a result, the system can not only more reliably retrieve textbook evidence, but also break down complex learning topics, supplement evidence, and reflect on and judge them, resulting in learning reports with a clearer structure, more complete evidence, and stronger traceability.

[0072] As one possible implementation, the intelligent agent workflow can share information such as user questions, document scope, subqueries, original evidence, filtered evidence, research rounds, report outlines, and citation mappings in the research state record, enabling each node to execute collaboratively under the same task state.

[0073] In some embodiments, the method further includes: In the process of generating the personalized learning report, the intelligent agent workflow is used to analyze the user's weak knowledge points based on the user's answers; During the generation of the personalized learning report, the intelligent agent workflow is used to adjust the structure and content focus of the personalized learning report based on the user's weak knowledge points, the learning topic, and the knowledge scope covered by the multimodal textbook knowledge base, and a textbook citation information viewing service is deployed for the personalized learning report.

[0074] In this embodiment, this application not only retrieves textbook evidence based on the learning topic, but also adjusts the structure and content focus of the report based on the weak knowledge points exposed in the pre-test.

[0075] Specifically, when generating personalized learning reports, the research module will share the weak knowledge points, learning topics, and knowledge base scope obtained from pre-test questions and learning analysis. This ensures that the report content focuses on explaining the knowledge points the user has not mastered, and the report structure will be adjusted to ensure that the content related to the weak knowledge points receives priority (e.g., by placing it at the beginning of the report), rather than simply providing a general summary of textbook information. Furthermore, a textbook citation information viewing service will be deployed for the personalized learning reports. For example, based on the correspondence between the report and the user's current knowledge weaknesses, corresponding textbook citation viewing functions will be deployed (e.g., highlighting the original text of the textbook based on PDF page rendering or screenshot display), allowing users to view relevant textbook content for their specific knowledge weaknesses, thereby improving the effectiveness of textbook-assisted learning.

[0076] In some embodiments, the method further includes: After generating the personalized learning report, based on the learning topic, the personalized learning report, historical dialogue records, and the multimodal textbook knowledge base, AI Tutor conducts multiple rounds of conversations with the user to provide multi-round tutoring, and retains textbook reference information each time it answers the user; After completing multiple rounds of tutoring, the intelligent agent workflow is used to generate post-test questions for the learning topic and record the user's answers. Using the intelligent agent workflow, by comparing the user's answers to pre-test questions and post-test questions, the system analyzes the user's mastery of knowledge points, score changes, and improvement of weaknesses, and generates a learning improvement analysis report. During the processing of the learning topic, the following data are stored in association: research session data, used to support the recovery of historical sessions and the viewing of report references; and teaching loop data, used to record teaching information throughout the entire process.

[0077] In this embodiment, considering that current systems are mostly limited to single question-and-answer sessions or content generation, lacking a closed-loop mechanism that identifies weak knowledge points based on the user's learning foundation, generates targeted learning materials, conducts multiple rounds of tutoring, and provides feedback on learning effectiveness through post-testing, it is difficult to continuously support the user's personalized learning process. Therefore, see [link to relevant documentation]. Figure 6The flowchart shown illustrates the implementation of an interactive teaching closed loop. This application designs a traceable interactive teaching closed loop method that integrates pre-testing, learning analysis, personalized learning reports, AI Tutor tutoring, post-test evaluation, and learning improvement analysis into a unified process. This process continuously records, diagnoses, tutors, and evaluates the user's learning status around the learning theme, forming a complete teaching closed loop. Furthermore, during the execution of the aforementioned process, the current learning theme and its related research session data and teaching closed loop data are associated and stored separately to support the implementation of system document management, citation backtracking, historical records, and teaching closed loop data association functions.

[0078] Specifically, after the learning report is generated, users can engage in multiple rounds of dialogue with AI Tutor based on the report's content. AI Tutor generates tutoring answers based on the current learning topic, report content, historical dialogue records, and search results from the textbook knowledge base.

[0079] In response to follow-up questions from users, textbook evidence can be retrieved again based on the multimodal textbook knowledge base, and textbook citation information can be retained in the answer to ensure that the multi-round tutoring process is always based on evidence.

[0080] After completing the learning report reading and AI Tutor tutoring, the intelligent agent workflow generates post-test questions around the same learning topic and records the user's answers. By comparing the results of the post-test and the pre-test, the system calculates the user's mastery of knowledge points, score changes, and improvement in weak areas, and generates a learning improvement analysis report. This report can serve as a basis for the user's subsequent review and also provide a reference for the system to continuously optimize personalized learning support.

[0081] Therefore, through the aforementioned closed-loop process, this application can generate targeted content based on the user's learning foundation and evaluate learning effectiveness through multiple rounds of tutoring and post-test feedback, thereby solving the problems of ordinary question-and-answer systems struggling to assess students' knowledge absorption and lacking process management. Simultaneously, this process can also provide a reference for adjusting subsequent teaching content.

[0082] As one possible implementation, the research session data may include learning reports, citation maps, evidence lists, subqueries, report outlines, and AI Tutor multi-turn dialogue messages, etc., to support the recovery of historical sessions and the viewing of report citations.

[0083] As one possible implementation method, the teaching closed-loop data may include learning topics, pre-test questions and answers, pre-test scores, weak knowledge points, learning analysis results, post-test questions and answers, post-test scores, learning improvement, learning effect summary, and the current teaching stage, etc., to fully record the entire process of teaching information from pre-testing, learning diagnosis, personalized learning to post-test evaluation.

[0084] As one possible implementation, in front-end interaction, citation numbers in reports or tutoring answers are rendered as clickable markers. After the user clicks on a citation, the back-end queries the corresponding knowledge block and returns information such as the textbook title path, page range, text summary, and original area coordinates; the front-end then combines PDF page rendering or screenshot display to highlight the original text location in the textbook.

[0085] As one possible implementation method, data such as pre-test questions and scores, weak knowledge points, learning analysis, learning reports, tutoring dialogue records, post-test questions and scores, and learning improvement analysis reports are stored in corresponding data tables, so that a complete learning activity can form a traceable record.

[0086] For example, see Figure 7 The overall architecture diagram of the educational intelligent agent system shown in this application provides an educational intelligent agent system that can effectively organize knowledge from multimodal teaching materials, stably retrieve teaching material evidence, generate traceable learning reports, and support a complete teaching loop. This system is not limited to calling a general large language model for question answering, but can realize a complete technical solution around multimodal knowledge organization of teaching material PDF files, teaching material evidence retrieval, in-depth research of intelligent agents, and user learning loop.

[0087] Specifically, the system mainly consists of a front-end application layer, a back-end service layer, and a data storage layer. The front-end application layer is responsible for presenting the learning process and learning space interfaces to users, such as displaying the knowledge base management page, report display page, original text tracing page, and conversation management page. The back-end service layer provides interfaces for textbook parsing, knowledge entry, hybrid retrieval, report generation, test assessment, tutoring dialogue, and data management. The data storage layer includes structured databases (such as PostgreSQL), vector databases (such as Milvus), and file storage.

[0088] Through the aforementioned layered architecture, the system integrates textbook management, knowledge retrieval, learning report generation, and interactive teaching processes under a unified technical framework. In one implementation, the system front-end is implemented using React and Vite, the back-end uses FastAPI to provide service interfaces, structured data is stored in PostgreSQL, and vector indexes are implemented using Milvus. It is understood that these specific technical components are only used to illustrate one implementation method and do not constitute a limitation on the scope of protection of this application.

[0089] This application focuses on system function testing around the core business chain of this invention, with a focus on verifying key functions such as knowledge base and textbook entry management, hybrid retrieval and learning report generation, interactive teaching loop, original text tracing, and historical conversation management to ensure they operate normally as expected. The tests used mathematics textbooks or note-style PDFs as knowledge base materials and conducted comprehensive verification through a combination of front-end page operations, back-end interface calls, and partial unit test results.

[0090] Knowledge base management and document import testing: This test primarily verifies functions such as knowledge base creation, PDF textbook uploading, asynchronous parsing, document status updates, and handling of abnormal uploads. During the test, after creating a knowledge base and uploading a mathematics PDF document on the knowledge base management page, the system generates an import task, completes document parsing, and displays the knowledge base name, document name, and document status on the page. Once the textbook parsing is complete, the document status displays as PARSED, indicating that the document has been successfully imported and can be used for subsequent retrieval and learning processes.

[0091] In abnormal scenario testing, when uploading non-PDF files, the system displays "Only PDF files are supported" and prevents the file from proceeding to the next stage of the data entry process; after deleting a document, the corresponding document record is removed from the knowledge base page. This demonstrates that the knowledge base management and document entry modules operate stably under both normal and abnormal input scenarios.

[0092] Hybrid retrieval and learning report generation test: This test aims to verify whether the system can complete mixed retrieval, evidence screening, structured learning report generation, and citation tag mapping after receiving learning topics. Before the test, the knowledge base had been imported with the "Math Notes" textbook PDF and 200 knowledge blocks were parsed. The test set two different learning topics: "The concept of indefinite integrals and the relationship between antiderivatives and indefinite integrals" and "Initial value problems of first-order differential equations and their geometric meaning." The test was limited to the local knowledge base and web search was not enabled to eliminate external knowledge interference. A brief description of the test groups is shown in Table 1.

[0093] Table 1. Introduction to the Test Group

[0094] The system automatically splits subqueries around the learning topic and performs vector retrieval and full-text retrieval separately. Then, it merges and sorts the results using the Reciprocal Rank Fusion (RRF) algorithm. Taking the "Indefinite Integral" test group as an example, the subqueries generated by the system include the definition of indefinite integrals, the definition and properties of antiderivatives, the relationship between indefinite integrals and antiderivatives, and comparisons with textbook examples, used to retrieve different types of evidence such as definitions, properties, and examples from the textbook. The subquery generation results for test group A (indefinite integrals) are shown in Table 2. A comparison of the end-to-end test results of the hybrid retrieval and report generation is shown below. Figure 8 As shown.

[0095] Table 2. Results of the subquery generated for Test Group A (indefinite integral)

[0096] Test results show that test group A obtained 10 candidate pieces of evidence in the first round of retrieval, of which 6 were retained after node filtering; the second round of supplementary retrieval added 2 more, ultimately retaining 6 valid pieces of evidence. Test group B generated 5 subqueries, collected 11 candidate pieces of evidence in the first round of retrieval, retained 7 after filtering, added 1 more in the second round, and ultimately retained 7 valid pieces of evidence. Subsequently, the Writer node generated a Markdown-formatted learning report based on the valid evidence and established a mapping relationship between report citations and textbook knowledge blocks through citation_map. The structured learning report generation effect (test group A) is as follows: Figure 9 As shown.

[0097] In the citation tracing example, citation [1] of test group A corresponds to "definition of indefinite integral" in the textbook, and citation [2] corresponds to "properties of indefinite integral". They come from different sections of the same chapter in the textbook, indicating that the retrieval results can cover the definition and properties under the same topic. In the relevant unit tests, all 5 tests of RRF fusion logic and Writer citation fallback logic passed, and 0 failed. Among them, the example of the citation tracing mapping relationship in the generated report is shown in Table 3.

[0098] Table 3. Example of Report Citation and Source Mapping Relationship

[0099] Interactive teaching closed-loop function test: This test is the core verification content of the interactive teaching closed-loop functional module. It aims to fully verify whether the complete interactive teaching closed loop, consisting of five major stages—pre-testing, weakness analysis, personalized learning reports, post-test evaluation, and learning improvement analysis—can operate smoothly according to its design logic, and to verify whether this complete teaching chain, covering the initial assessment of learning progress to the final review of learning outcomes, meets the functional design requirements. This test selected the learning topic "The Definition of the Derivative and the Relationship with Limits" as the test object. The entire test was conducted within an existing knowledge base environment with complete content reserves and a pre-built setup. It strictly followed the preset process of the teaching closed loop, sequentially completing the three core process stages: pre-test answering, personalized learning report generation, and post-test answering.

[0100] The test results showed that the system generated 8 pre-test questions in 49.6 seconds. After submitting the pre-test answers, the system completed the evaluation in 29.4 seconds, scoring 43 points with 3 correct answers. It also identified 3 weak knowledge points: "definition of the derivative," "necessary and sufficient conditions for one-sided derivatives and differentiability," and "the relationship between the definition of the derivative and limits." Subsequently, the system generated a personalized learning report based on these weak knowledge points. The report was 6437 characters long, included 2 original text citations, and took 154.1 seconds to generate. After reading the report, the system proceeded to the post-test stage, generating 7 test questions targeting the weak knowledge points and returning a comparison of the pre- and post-test results upon submission.

[0101] The test interface showed a pre-test score of 43 and a post-test score of 29, a decrease of 14 points. The system also provided a corresponding summary, pointing out significant weaknesses in areas such as one-sided derivatives and differentiability, and the relationship between limits and function behavior. Although this test did not show an improvement in scores, the system's functional workflow verified its ability to complete question generation, answer evaluation, weakness identification, learning report generation, post-test evaluation, and result summary. The interactive teaching closed-loop test results are as follows: Figure 10 As shown.

[0102] Original source tracing and historical conversation management test: This test covers two parts: original text tracing and historical session management. The original text tracing function is triggered when a user clicks on a citation number in the learning report. The system will then display a pop-up source card showing the chapter path, page number, source type, and content preview of the corresponding knowledge block. Users can also jump to the corresponding location in the PDF textbook using the "View Original Text" button. The original text tracing source citation card is shown below. Figure 11 As shown.

[0103] The test results show that after clicking the reference mark [1] in the main text of the report, the system successfully pops up the reference card, showing that its source is "Chapter 4 Indefinite Integrals > 4.1 Concept and Properties of Indefinite Integrals > 4.1.1 Concept of Antiderivative and Indefinite Integral (Introduction)", located on page 34 of the textbook, with the source type marked as "Local Knowledge Base", and a content preview is attached. This indicates that the system has been able to construct a complete tracing path from the reference in the learning report to the original text of the textbook.

[0104] Regarding historical session management, the system features a research history list on the left panel, used to record the title, completion status, and creation time of each learning session. During testing, the history list displayed seven session records, including "Completed" and "In Progress" statuses, covering learning topics such as "the definition of the derivative and its relationship to limits" and "the concept of indefinite integrals and the relationship between antiderivatives and indefinite integrals." The titles and status labels of each record were displayed correctly. The research history session list is as follows: Figure 12 As shown in Table 4, the test results for original text tracing and historical conversation management are as follows.

[0105] Table 4. Test Results of Original Text Tracing and Historical Conversation Management

[0106] Based on the same inventive concept, embodiments of this application also provide an educational intelligent question-answering system based on multimodal knowledge parsing, the system comprising: The multimodal textbook knowledge organization module is used to respond to received textbook materials uploaded by users, perform structured parsing on the textbook materials to obtain parsing results, the parsing results including: multimodal textbook content information, and textbook layout and position information; and generate semantic descriptions for each first-type element based on its respective textbook content information and context information in the parsing results, wherein different types of elements correspond to different modalities, and each first-type element is a non-text element; and organize the parsing results and the semantic descriptions of each first-type element into a unified content element list; and perform structured segmentation of the unified content element list to obtain multiple knowledge blocks; and construct a multimodal textbook knowledge base based on the multiple knowledge blocks; The textbook evidence retrieval module is used to respond to the detection of the query text to be retrieved associated with the user's question, and to retrieve the various knowledge blocks associated with the query text based on the multimodal textbook knowledge base; The answer generation module is used to generate an answer containing textbook citation information based on the knowledge blocks associated with all the retrieved query texts after completing the retrieval of all the query texts associated with the user's question.

[0107] Optionally, the multimodal textbook knowledge base includes a structured database and a vector database; the multimodal textbook knowledge organization module is further used to write each knowledge block and its source metadata into the structured database, and to write each knowledge block into the vector database; The textbook evidence retrieval module is further configured to process the query text into a query vector using an embedding model, and retrieve semantically similar knowledge blocks from the vector database based on the query vector to obtain vector retrieval results; and to perform keyword retrieval on the query text based on the structured database to recall query terms associated with the query text, as well as knowledge blocks that match at least one of chapter titles, formula titles, and proper nouns to obtain full-text retrieval results; and to determine the various knowledge blocks associated with the query text based on the vector retrieval results and the full-text retrieval results.

[0108] Optionally, the textbook evidence retrieval module is further configured to determine the comprehensive ranking score of each retrieved knowledge block based on its ranking in the vector retrieval results and the full-text retrieval results; and to filter all retrieved knowledge blocks based on the comprehensive ranking score to obtain the knowledge blocks associated with the query text.

[0109] Optionally, the multimodal textbook knowledge organization module is also used to perform the following steps: For each image type element, a semantic description of the image type element is generated based on the image title, screenshot, and surrounding text in the parsing results. For each table-type element, a semantic description of the table-type element is generated based on the table screenshot, table title, table content, and surrounding text in the parsing results. For table-type elements that span multiple pages, before generating semantic descriptions, the information associated with the table-type elements in the parsing results is merged in advance based on the page-spanning relationship between adjacent pages.

[0110] Optionally, the multimodal textbook knowledge organization module is further configured to combine the information associated with each first type element in the unified content element list into multimodal knowledge blocks based on the title tree and chapter boundaries, wherein each first type element includes image type elements and / or table type elements; and to merge the information associated with each second type element in the unified content element list according to the title tree and chapter boundaries, based on semantic continuity and preset length constraints, to obtain each text knowledge block, wherein each second type element includes text type elements and / or formula type elements.

[0111] Optionally, the multimodal textbook knowledge organization module is further used to construct the title tree through the following steps: Based on the title numbering, font style, page position, and text pattern of the teaching materials, as well as the preset table of contents noise filtering rules, the chapter levels of the teaching materials are identified. The preset table of contents noise filtering rules include: title filtering rules based on at least one of the following: numbering pattern, page number position, table of contents items, cover text, and miscellaneous chart numbers. A title tree is constructed based on the chapter hierarchy of the textbook. Each node in the title tree corresponds to a chapter or section. The node records the title name, level, start and end page numbers, parent-child relationship, and title path of the corresponding chapter or section.

[0112] Optionally, the device further includes a teaching assistance module for performing the following steps using an agent workflow in response to receiving a user question containing a learning topic: Pre-test questions are generated based on the learning topic, and the user's answers are recorded; Task planning is performed on the learning topic to generate query texts corresponding to each sub-query, and evidence reflection and supplementary retrieval are performed on the textbook evidence units associated with each query text; The answer generation module is further configured to complete the target content of each knowledge block to obtain textbook evidence units for the agent workflow. The target content includes at least one of the following: title path, page range, block type, text summary, and source information. Using the agent workflow, based on the user's answer and all textbook evidence units associated with the learning topic, a personalized learning report and cited references are generated. The cited references record the correspondence between the evidence numbers cited in the personalized learning report and multiple preset dimension information, including knowledge block number, document number, title path, page range, block type, and content summary. Based on the personalized learning report and cited references, an answer carrying textbook citation information is generated.

[0113] Optionally, the answer generation module is further configured to, during the generation of the personalized learning report, utilize the intelligent agent workflow to analyze the user's weak knowledge points based on the user's answer results; and during the generation of the personalized learning report, utilize the intelligent agent workflow to adjust the structure and content focus of the personalized learning report based on the user's weak knowledge points, the learning topic, and the knowledge scope covered by the multimodal textbook knowledge base, and deploy a textbook citation information viewing service for the personalized learning report.

[0114] Optionally, the teaching assistance module is further configured to, after generating the personalized learning report, conduct multi-round tutoring sessions with the user using AI Tutor based on the learning topic, the personalized learning report, historical dialogue records, and the multimodal textbook knowledge base, and retain textbook citation information each time the user is answered; after completing the multi-round tutoring, generate post-test questions for the learning topic using the intelligent agent workflow and record the user's answers; and use the intelligent agent workflow to analyze the user's knowledge mastery, score changes, and improvement of weaknesses by comparing the user's answers to the pre-test questions and post-test questions, and generate a learning improvement analysis report; and during the processing around the learning topic, associate and store the following data respectively: research conversation data, used to support the recovery of historical conversations and viewing of report citations; and teaching closed-loop data, used to record teaching information throughout the entire process.

[0115] The educational intelligent question-answering system based on multimodal knowledge parsing provided in this application embodiment can achieve... Figure 1 The various processes implemented in the example of the educational intelligent question-answering method based on multimodal knowledge parsing shown here achieve the same technical effect, and will not be repeated here to avoid duplication.

[0116] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described embodiments of the educational intelligent question-answering method based on multimodal knowledge parsing, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0117] The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0118] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described embodiments of the educational intelligent question-answering method based on multimodal knowledge parsing, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0119] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0120] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0122] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An educational intelligent question-answering method based on multimodal knowledge parsing, characterized in that, The method includes: In response to receiving textbook materials uploaded by a user, the textbook materials are subjected to structured parsing to obtain parsing results, which include: multimodal textbook content information, and textbook layout and location information; Based on the textbook content information and context information of each first type element in the parsing results, a semantic description of each first type element is generated, wherein different types of elements correspond to different modalities, and each first type element is a non-text type element; The parsing results and the semantic descriptions of each of the first type elements are organized into a unified content element list; The unified content element list is structured and divided into multiple knowledge blocks; Based on the aforementioned knowledge blocks, a multimodal textbook knowledge base is constructed; In response to detecting the query text to be retrieved associated with the user's question, the system retrieves the various knowledge blocks associated with the query text based on the multimodal textbook knowledge base. After retrieving all the query texts associated with the user's question, an answer containing textbook citation information is generated based on the knowledge blocks associated with all the retrieved query texts.

2. The method according to claim 1, characterized in that, The multimodal textbook knowledge base includes a structured database and a vector database; the construction of the multimodal textbook knowledge base based on the multiple knowledge blocks includes: Each knowledge block and its source metadata are written into the structured database, and each knowledge block is written into the vector database; The step of retrieving various knowledge blocks associated with the query text based on the multimodal textbook knowledge base includes: The query text is processed into a query vector using an embedding model, and semantically similar knowledge blocks are retrieved from the vector database based on the query vector to obtain vector retrieval results; Based on the structured database, keyword retrieval is performed on the query text to recall the query terms associated with the query text, as well as knowledge blocks that match at least one of chapter titles, formula titles, and proper nouns, to obtain full-text retrieval results; Based on the vector retrieval results and the full-text retrieval results, the knowledge blocks associated with the query text are determined.

3. The method according to claim 2, characterized in that, The step of determining the knowledge blocks associated with the query text based on the vector retrieval results and the full-text retrieval results includes: For each retrieved knowledge block, a comprehensive ranking score is determined based on the ranking of the knowledge block in the vector retrieval results and the full-text retrieval results, respectively. Based on the comprehensive ranking score, all retrieved knowledge blocks are filtered to obtain the knowledge blocks associated with the query text.

4. The method according to claim 1, characterized in that, The step of generating semantic descriptions for each of the first-type elements based on their respective textbook content information and context information in the parsing results includes: For each image type element, a semantic description of the image type element is generated based on the image title, screenshot, and surrounding text in the parsing results. For each table-type element, a semantic description of the table-type element is generated based on the table screenshot, table title, table content, and surrounding text in the parsing results. The method further includes: For table-type elements that span multiple pages, before generating semantic descriptions, the information associated with the table-type elements in the parsing results is merged in advance based on the page-spanning relationship between adjacent pages.

5. The method according to claim 1, characterized in that, The unified content element list is structured and segmented to obtain multiple knowledge blocks, including: For the information associated with each first type element in the unified content element list, the information is combined into various multimodal knowledge blocks according to the title tree and chapter boundaries. Each first type element includes image type elements and / or table type elements. For the information associated with each second type element in the unified content element list, the information is merged according to semantic continuity and preset length constraints based on the title tree and chapter boundaries to obtain each text knowledge block. Each second type element includes text type elements and / or formula type elements.

6. The method according to claim 5, characterized in that, The title tree is constructed through the following steps: Based on the title numbering, font style, page position, and text pattern of the teaching materials, as well as the preset table of contents noise filtering rules, the chapter levels of the teaching materials are identified. The preset table of contents noise filtering rules include: title filtering rules based on at least one of the following: numbering pattern, page number position, table of contents items, cover text, and miscellaneous chart numbers. A title tree is constructed based on the chapter hierarchy of the textbook. Each node in the title tree corresponds to a chapter or section. The node records the title name, level, start and end page numbers, parent-child relationship, and title path of the corresponding chapter or section.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: In response to receiving a user question containing a learning topic, the following steps are executed using the agent workflow: Pre-test questions are generated based on the learning topic, and the user's answers are recorded; Task planning is performed on the learning topic to generate query texts corresponding to each sub-query, and evidence reflection and supplementary retrieval are performed on the textbook evidence units associated with each query text; The step of generating an answer carrying textbook citation information based on the knowledge blocks associated with all retrieved query texts includes: The target content of each knowledge block is completed to obtain a textbook evidence unit for the intelligent agent workflow. The target content includes at least one of the following: title path, page number range, block type, text summary and source information. Using the intelligent agent workflow, a personalized learning report and cited references are generated based on the user's answer and all textbook evidence units associated with the learning topic. The cited references are used to record the correspondence between the evidence numbers cited in the personalized learning report and multiple preset dimension information, including knowledge block number, document number, title path, page range, block type, and content summary. Based on the personalized learning report and cited references, generate answers that include textbook citation information.

8. The method according to claim 7, characterized in that, The method further includes: In the process of generating the personalized learning report, the intelligent agent workflow is used to analyze the user's weak knowledge points based on the user's answers; During the generation of the personalized learning report, the intelligent agent workflow is used to adjust the structure and content focus of the personalized learning report based on the user's weak knowledge points, the learning topic, and the knowledge scope covered by the multimodal textbook knowledge base, and a textbook citation information viewing service is deployed for the personalized learning report.

9. The method according to claim 7, characterized in that, The method further includes: After generating the personalized learning report, based on the learning topic, the personalized learning report, historical dialogue records, and the multimodal textbook knowledge base, AI Tutor conducts multiple rounds of conversations with the user to provide multi-round tutoring, and retains textbook reference information each time it answers the user; After completing multiple rounds of tutoring, the intelligent agent workflow is used to generate post-test questions for the learning topic and record the user's answers. Using the intelligent agent workflow, by comparing the user's answers to pre-test questions and post-test questions, the system analyzes the user's mastery of knowledge points, score changes, and improvement of weaknesses, and generates a learning improvement analysis report. During the processing of the learning topic, the following data are stored in association: research session data, used to support the recovery of historical sessions and the viewing of report references; and teaching loop data, used to record teaching information throughout the entire process.

10. An educational intelligent question-answering system based on multimodal knowledge parsing, characterized in that, The system includes: The multimodal textbook knowledge organization module is used to respond to received textbook materials uploaded by users, perform structured parsing on the textbook materials to obtain parsing results, the parsing results including: multimodal textbook content information, and textbook layout and position information; and generate semantic descriptions for each first-type element based on its respective textbook content information and context information in the parsing results, wherein different types of elements correspond to different modalities, and each first-type element is a non-text element; and organize the parsing results and the semantic descriptions of each first-type element into a unified content element list; and perform structured segmentation of the unified content element list to obtain multiple knowledge blocks; and construct a multimodal textbook knowledge base based on the multiple knowledge blocks; The textbook evidence retrieval module is used to respond to the detection of the query text to be retrieved associated with the user's question, and to retrieve the various knowledge blocks associated with the query text based on the multimodal textbook knowledge base; The answer generation module is used to generate an answer containing textbook citation information based on the knowledge blocks associated with all the retrieved query texts after completing the retrieval of all the query texts associated with the user's question.

Images