A teaching question-answering method and system based on a multi-modal knowledge base

Abstract

The present application relates to the technical field of computer big data, and discloses a teaching question and answer method and system based on a multi-modal knowledge base, aiming at solving the problem of inaccurate semantic recognition and single response form of the existing teaching question and answer system due to the lack of subject pertinence. The subject category of the user's question is identified, a semantic recognition strategy matched with the subject knowledge characteristics is called, the question intention and associated elements are accurately analyzed, the matching content unit is searched in the multi-modal knowledge base according to the analysis result, the most suitable response modal format is dynamically determined according to the subject category and question elements, and finally the searched content is intelligently arranged and packaged according to the format to generate and output a structured multi-modal answer. The present application realizes the precision and personalization of cross-subject question and answer by deeply integrating the subject characteristics into the whole process of semantic recognition and response generation, and significantly improves the accuracy and effectiveness of teaching assistance.

Description

Technical Field

[0001] This application relates to the field of computer big data technology, and in particular to a teaching question-and-answer method and system based on a multimodal knowledge base. Background Technology

[0002] Traditional question-and-answer systems in teaching typically interact with general natural language processing models and structured knowledge bases, aiming to provide users with standardized answers. These systems often overlook the fundamental differences in knowledge representation, logical deduction, and cognitive presentation across different disciplines. For example, mathematics emphasizes formula derivation and step-by-step calculations, humanities focus on outlining the context and comparing viewpoints, while arts rely on visual examples and style analysis. Existing methods use the same semantic parsing and response generation strategy to handle questions across all disciplines, leading to decreased parsing accuracy when dealing with specialized questions from specific disciplines. The generated answers are often monotonous and fail to align with the unique thinking and expression habits of that discipline.

[0003] Furthermore, some existing teaching systems that attempt to introduce multimodal content, while integrating various media resources such as text and images, suffer from significant deficiencies in the targeted use and adaptive organization of multimodal resources in their question-and-answer mechanisms. These systems typically fail to establish an inherent connection between subject-specific characteristics and multimodal presentation formats; their responses merely list or juxtapose relevant multimedia materials, lacking in-depth content integration and structured arrangement based on subject-specific logic. This deficiency makes the responses appear loose, provides weak instructional guidance, and fails to effectively explain the core concepts and complex processes of different disciplines through the most appropriate media combinations and narrative logic.

[0004] Therefore, existing question-answering methods in teaching technology lack subject-sensitive parsing strategies in the semantic recognition stage and lack the ability to dynamically adapt to multimodal formats that match the subject content in the response generation stage. This results in limitations in the accuracy, applicability, and teaching effectiveness of the generated responses, making it difficult to meet the demand for precise and personalized knowledge services in interdisciplinary intelligent teaching assistance. There is an urgent need for a novel question-answering method that can automatically adapt different semantic recognition strategies and response modal formats based on the subject of the question. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A teaching question-answering method based on a multimodal knowledge base includes the following steps:

[0007] S1: Receive the user's original question-and-answer request;

[0008] S2: Identify the subject category of the original question-and-answer request and determine its target subject category;

[0009] S3: Based on the target subject category, call the corresponding question-and-answer semantic recognition strategy from the preset subject feature library. The question-and-answer semantic recognition strategy defines semantic parsing rules and key element extraction priorities that match the knowledge expression characteristics of the subject.

[0010] S4: Using the invoked question-and-answer semantic recognition strategy, perform semantic parsing on the original question-and-answer request to identify the core question intent and multiple related semantic elements;

[0011] S5: Based on the core question intent and associated semantic elements, perform retrieval and matching in a pre-built multimodal knowledge base to obtain multiple basic content units associated with the core question intent. The content units stored in the multimodal knowledge base are marked with content type tags and subject category tags.

[0012] S6: Determine the target modal format of the response based on the target subject category and the associated semantic elements; wherein, the target modal format defines the combination rules of content organization structure and media presentation form;

[0013] S7: Based on the target modal format, the retrieved multiple basic content units are arranged and formatted to generate multimodal response content that meets the requirements of the target modal format;

[0014] S8: Output the multimodal response content in response to the user's original question and answer request.

[0015] Furthermore, the subject category identification process in step S2 includes:

[0016] S21: Extract text features and / or symbol features from the original question-and-answer request;

[0017] S22: Match the text features and / or symbol features with the predefined keyword sets and concept patterns of each subject category in the subject feature library;

[0018] S23: Determine the target subject category to which the original question and answer request belongs based on the subject category keyword set and concept pattern with the highest matching degree.

[0019] Furthermore, step S4 employs a question-and-answer semantic recognition strategy to perform semantic parsing on the original question-and-answer request, including:

[0020] S41: Perform syntactic structure analysis and dependency relation analysis on the original question-answering request according to the semantic parsing rules defined in the question-answering semantic recognition strategy;

[0021] S42: Based on the results of the syntactic structure analysis and dependency relation analysis, extract priorities according to the key elements defined in the question-answering semantic recognition strategy, and identify and label the core action subject, core action object, core limiting conditions and problem-solving objectives in the original question-answering request;

[0022] S43: Map the identified core action subject, core action object, core limiting conditions, and problem-solving objective into a structured representation of the core problem intent;

[0023] S44: Extract auxiliary descriptive information, contextual indicators, or comparison relation words from the original question-and-answer request, excluding the core elements, as the associated semantic elements.

[0024] Furthermore, step S5, which involves retrieval and matching in a pre-built multimodal knowledge base, includes:

[0025] S51: Based on the core question intent, construct the first search condition, and initially filter out a set of candidate content units in the multimodal knowledge base whose content type tags and subject category tags both match the core question intent;

[0026] S52: Based on the associated semantic elements, construct a second search condition, the second search condition including requirements for the details, attributes or relationships indicated by the associated semantic elements;

[0027] S53: Based on the second retrieval condition, perform secondary screening and matching in the candidate content unit set, and calculate the semantic relevance between each candidate content unit and the second retrieval condition;

[0028] S54: Based on the semantic relevance, select multiple content units with the highest relevance from the candidate content unit set as the basic content units.

[0029] Furthermore, in step S6, determining the target modal format of the response is specifically as follows:

[0030] S61: Based on the target subject category, search for one or more candidate modal formats supported by the subject category from the preset subject-modal mapping table;

[0031] S62: Analyze the associated semantic elements to determine whether they contain implicit requirements for specific presentation forms, interaction methods, or logical display order;

[0032] S63: Combining the candidate modal formats and the implicit requirements, select or combine the candidate modal formats to generate the most suitable modal format as the target modal format.

[0033] Furthermore, the method also includes:

[0034] In the subject-modal mapping table, the candidate modal formats associated with science subjects include step-by-step derivation, formula and chart combination, and dynamic simulation demonstration; the candidate modal formats associated with humanities subjects include timeline narrative, comparative analysis table, and structured outline; and the candidate modal formats associated with arts subjects include work example display, style comparison atlas, and creative process decomposition.

[0035] Furthermore, step S7 involves arranging and formatting the retrieved basic content units, including:

[0036] S71: Parse the content organization structure rules of the target modal format to determine the arrangement order, hierarchical relationship and logical connection method of the multiple basic content units;

[0037] S72: Parse the media presentation rules of the target modal format, and assign corresponding presentation controls or display templates to the text, image, audio or video content in the multiple basic content units;

[0038] S73: According to the content organization structure rules and media presentation format rules, the multiple basic content units and their metadata are encapsulated into response data packets that can be independently rendered or played.

[0039] Furthermore, the method also includes an interaction optimization step following step S8:

[0040] S9: After outputting the multimodal response content, receive feedback or follow-up questions from the user regarding the multimodal response content;

[0041] S10: Analyze the feedback or follow-up questions to determine whether they constitute a deepening or correction of the core question's intent or a supplement to the related semantic elements;

[0042] S11: If it is determined to be a deepening, correction or supplement, the feedback or follow-up question will be used as a new related semantic element. Combined with the original core question intent and the target subject category, a new round of semantic parsing, knowledge retrieval and content generation process starting from step S4 will be triggered to generate updated multimodal response content.

[0043] S12: Output the updated multimodal response content.

[0044] Furthermore, the method also includes:

[0045] The multimodal knowledge base is constructed as follows:

[0046] The knowledge content is divided according to subject categories;

[0047] Within each subject category, the knowledge content is categorized according to the content type of concepts, principles, cases, and exercises;

[0048] Each knowledge content unit is labeled with one or more semantic recognition strategy identifiers applicable to it and one or more recommended modal format identifiers.

[0049] According to a second aspect of the present invention, the present invention claims protection for a teaching question-answering system based on a multimodal knowledge base, comprising:

[0050] One or more processors;

[0051] A memory that stores one or more programs, which, when executed by one or more processors, enable the processors to implement the teaching question-and-answer method based on a multimodal knowledge base.

[0052] This invention discloses a teaching question-and-answer method and system based on a multimodal knowledge base, aiming to solve the problems of inaccurate semantic recognition and monotonous response formats in existing teaching question-and-answer systems due to a lack of subject-specificity. The method identifies the subject category of the user's question; invokes a semantic recognition strategy matching the knowledge characteristics of that subject to accurately parse the question's intent and related elements; retrieves matching content units from the multimodal knowledge base based on the parsing results; dynamically determines the most suitable response modality format according to the subject category and question elements; and finally, intelligently arranges and encapsulates the retrieved content according to this format to generate and output a structured multimodal answer. This invention deeply integrates subject characteristics into the entire process of semantic recognition and response generation, achieving precise and personalized cross-disciplinary question-and-answer, significantly improving the accuracy and effectiveness of teaching assistance. Attached Figure Description

[0053] Figure 1 A flowchart illustrating the workflow of a teaching question-and-answer method based on a multimodal knowledge base, as claimed in an embodiment of the present invention.

[0054] Figure 2 The second flowchart of a teaching question-answering method based on a multimodal knowledge base, as claimed in an embodiment of the present invention, is shown below.

[0055] Figure 3 A third workflow diagram of a teaching question-answering method based on a multimodal knowledge base, as claimed in this embodiment of the invention;

[0056] Figure 4 The fourth workflow diagram of a teaching question-and-answer method based on a multimodal knowledge base, as claimed in this embodiment of the invention. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include multiple such features. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications in the embodiments of this application, such as up, down, left, right, front, back, etc., are only used to explain the relative positional relationships and movements between components in a specific orientation as shown in the accompanying drawings. If the specific orientation changes, the directional indications will change accordingly. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0059] References to embodiments herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in multiple embodiments of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0060] According to a first embodiment of the present invention, the present invention claims protection for a teaching question-answering method based on a multimodal knowledge base, referring to... Figure 1 This includes the following steps:

[0061] S1: Receive the user's original question-and-answer request;

[0062] S2: Identify the subject category of the original question-and-answer request and determine its target subject category;

[0063] S3: Based on the target subject category, call the corresponding question-and-answer semantic recognition strategy from the preset subject feature library. The question-and-answer semantic recognition strategy defines semantic parsing rules and key element extraction priorities that match the knowledge expression characteristics of the subject.

[0064] S4: Using the invoked question-and-answer semantic recognition strategy, perform semantic parsing on the original question-and-answer request to identify the core question intent and multiple related semantic elements;

[0065] S5: Based on the core question intent and associated semantic elements, perform retrieval and matching in a pre-built multimodal knowledge base to obtain multiple basic content units associated with the core question intent. The content units stored in the multimodal knowledge base are marked with content type tags and subject category tags.

[0066] S6: Based on the target subject category and the associated semantic elements, determine the target modal format of the response, wherein the target modal format defines the combination rules of content organization structure and media presentation form;

[0067] S7: Based on the target modal format, the retrieved multiple basic content units are arranged and formatted to generate multimodal response content that meets the requirements of the target modal format;

[0068] S8: Output the multimodal response content in response to the user's original question and answer request.

[0069] In this embodiment, S1 receives the user's original question-and-answer request. This request is typically a natural language question or phrase submitted by the user through a text input interface, and the system captures the complete string of the request through an interface.

[0070] In step S2, the original question-and-answer request is identified by subject category to determine its target subject category. This process is accomplished by analyzing the terminology, concepts, and question expression patterns in the request. The system compares the request content with a predefined subject classification system, which covers multiple major subject areas such as mathematics, physics, history, literature, and art. The identification mechanism relies not only on keyword matching but also on the combination of terms and typical question structures. For example, a request containing chemical equations and descriptions of reaction conditions will be classified as chemistry, while a request involving causal analysis of historical events will be classified as history. The identification result is a clear subject category identifier.

[0071] In step S3, based on the target subject category, the corresponding question-answering semantic recognition strategy is invoked from a preset subject feature library. The subject feature library is a configuration file or database that stores the mapping relationship between different subject categories and their dedicated semantic recognition strategies. Each strategy is a set of rules or parameters that defines how to parse questions for that subject. For example, a recognition strategy for mathematics might emphasize extracting target words such as numerical values, operators, variables, and terms like "solve" and "proof"; a recognition strategy for literature might focus on work titles, authors, characters, plot elements, and intent words like "analysis" and "appreciation."

[0072] In step S4, the invoked question-and-answer semantic recognition strategy is used to perform semantic parsing on the original question-and-answer request. This step applies the rules in the strategy to deeply analyze sentence components. The system identifies the core of the user's intent, i.e., what knowledge they mainly want to acquire or what problem they want to solve, and extracts auxiliary information, conditions, or objects related to the core intent; these are collectively referred to as associated semantic elements. For example, in the physics problem of calculating the acceleration of an object subjected to friction on an inclined plane, the core problem intent is to calculate acceleration, and the associated semantic elements include the inclined plane, friction, and the object.

[0073] In S5, based on the core question intent and related semantic elements, retrieval and matching are performed in a pre-built multimodal knowledge base. This multimodal knowledge base stores knowledge content units organized in various media formats such as text descriptions, formulas, charts, diagrams, audio clips, and animation clips. Each unit has metadata tags indicating its subject and content type. The retrieval process first uses the core question intent as the primary query condition to filter out content units of relevant subjects and types. Then, it uses related semantic elements for refined filtering and matching to find the specific knowledge unit that best fits the question context.

[0074] In step S6, the target modal format for the response is determined based on the target subject category and the associated semantic elements. The modal format defines the organizational structure and presentation of the final answer. The system determines the general direction based on the characteristics of the subject, and then selects or synthesizes the final format in combination with the specific needs of the question. For example, a question about geometric proofs may be best suited to a step-by-step derivation diagram format, while a question about poetry appreciation may be more suitable for a text excerpt with commentary and comparison format.

[0075] In step S7, based on the target modal format, the system performs content arrangement and format encapsulation on the retrieved at least one basic content unit. The system sorts, combines, labels, and packages the found knowledge content units according to the requirements of the target format. For example, if the target format is a timeline narrative, the system will arrange relevant historical event content units in chronological order and insert them into a timeline graphic template to form a coherent narrative flow.

[0076] In step S8, the multimodal response content is output to respond to the user's original question-and-answer request. The encapsulated response data packet is rendered and displayed to the user through the user interface, and may include text paragraphs, images, structured lists, or interactive controls.

[0077] Furthermore, referring to Figure 2 The subject category identification process in step S2 includes:

[0078] S21: Extract text features and / or symbol features from the original question-and-answer request;

[0079] S22: Match the text features and / or symbol features with the predefined keyword sets and concept patterns of each subject category in the subject feature library;

[0080] S23: Determine the target subject category to which the original question and answer request belongs based on the subject category keyword set and concept pattern with the highest matching degree.

[0081] In this embodiment, step S21 extracts textual and / or symbolic features from the original question-and-answer request. Textual features include, but are not limited to, subject-specific terminology such as the law of cosines, redox reactions, and high-frequency content words. Symbolic features include mathematical symbols such as ∑ and ∫, chemical formulas such as H₂O, and specific punctuation patterns. The system extracts these features through word segmentation, part-of-speech tagging, and pattern matching.

[0082] In step S22, the text features and / or symbolic features are matched with predefined keyword sets and concept patterns for each subject category in the subject feature library. The subject feature library maintains a feature set for each subject, including a list of core keywords and typical question patterns. The matching process calculates the overlap and matching strength between the extracted features and the feature sets of each subject. For example, a question containing words like "function," "derivative," and "monotonicity" has a high match with the keyword set of the mathematics subject; a question containing "Meiji Restoration," "background," and "impact" matches the concept patterns of the history subject.

[0083] In step S23, the target subject category of the original question-and-answer request is determined based on the keyword set and conceptual pattern of the subject category with the highest matching degree. The system employs a scoring mechanism to quantify the matching degree of each subject and selects the subject with the highest total score as the target subject category. If scores are similar or conflicting, the system may activate preset priority rules or perform more complex contextual analysis to make a decision, ultimately outputting a definite subject label.

[0084] Furthermore, referring to Figure 3 Step S4 employs a question-and-answer semantic recognition strategy to perform semantic parsing on the original question-and-answer request, including:

[0085] S41: Perform syntactic structure analysis and dependency relation analysis on the original question-answering request according to the semantic parsing rules defined in the question-answering semantic recognition strategy;

[0086] S42: Based on the results of the syntactic structure analysis and dependency relation analysis, extract priorities according to the key elements defined in the question-answering semantic recognition strategy, and identify and label the core action subject, core action object, core limiting conditions and problem-solving objectives in the original question-answering request;

[0087] S43: Map the identified core action subject, core action object, core limiting conditions, and problem-solving objective into a structured representation of the core problem intent;

[0088] S44: Extract auxiliary descriptive information, contextual indicators, or comparison relation words from the original question-and-answer request, excluding the core elements, as the associated semantic elements.

[0089] In this embodiment, in step S41, the original question-and-answer request is analyzed syntactically and in terms of dependency relations according to the semantic parsing rules defined in the question-and-answer semantic recognition strategy. The system applies grammatical and dependency relation analysis rules suitable for the target subject to analyze the subject-verb-object structure, attributive, adverbial, and complement components of the sentence, as well as the relationships between words such as modification, verb-object, and subject-verb. For example, for science questions, the focus is on analyzing the relationship between conditional adverbial clauses and the main clause; for humanities questions, the focus is on analyzing the complex sentence relationships introduced by conjunctions such as parallelism and contrast.

[0090] In step S42, based on the results of the syntactic structure analysis and dependency relation analysis, the extraction priority is determined according to the key elements defined in the question-and-answer semantic recognition strategy. This process identifies and labels the core action subject, core action object, core limiting conditions, and problem-solving objective in the original question-and-answer request. The extraction priority varies across different disciplines. In physics problems, the priority might be: first extract the physical quantity object, then extract the process or state conditions, and finally extract the desired quantity objective. In historical problems, the priority might be: first extract the historical event or person / subject / object, then extract the time, place, and causal conditions, and finally extract the angle of inquiry, such as cause, significance, or process.

[0091] In step S43, the identified core action subject, core action object, core limiting conditions, and problem-solving objective are mapped to a structured representation of the core problem intent. The discrete elements extracted in the previous step are organized according to a pre-defined intent expression framework. For example, a logical structure of action solving / analysis / comparison… + object + [condition] is formed. This structured representation serves as a precise query blueprint for subsequent retrieval of the knowledge base.

[0092] In S44, auxiliary descriptive information, contextual indicators, or comparative relational words, excluding the core elements, are extracted from the original question-and-answer request and used as the associated semantic elements. These elements provide background or refinement requirements for the core intent. For example, if the core intent is to explain photosynthesis, associated semantic elements might include using plain language, comparing it to respiration, or providing a diagram. The system will identify phrases such as "in the manner of," "compared to," or "detailed explanation of the part," and convert their content into actionable retrieval or format control parameters.

[0093] Furthermore, referring to Figure 4 Step S5, which involves retrieval and matching in a pre-built multimodal knowledge base, includes:

[0094] S51: Based on the core question intent, construct the first search condition, and initially filter out a set of candidate content units in the multimodal knowledge base whose content type tags and subject category tags both match the core question intent;

[0095] S52: Based on the associated semantic elements, construct a second search condition, the second search condition including requirements for the details, attributes or relationships indicated by the associated semantic elements;

[0096] S53: Based on the second retrieval condition, perform secondary screening and matching in the candidate content unit set, and calculate the semantic relevance between each candidate content unit and the second retrieval condition;

[0097] S54: Based on the semantic relevance, select multiple content units with the highest relevance from the candidate content unit set as the basic content units.

[0098] In this embodiment, step S51 constructs a first search condition based on the structured representation of the core question intent. This condition mainly includes the core actions and main objects in the intent, combined with their subject category tags and possible content type tags such as concepts, principles, and examples. Using this condition, a preliminary query is performed on the multimodal knowledge base to obtain a relatively broad set of candidate content units. All units in this set are thematically related to the core question intent.

[0099] In step S52, a second search condition is constructed based on the aforementioned related semantic elements. The second search condition aims to express requirements for details, attributes, or specific relationships within the knowledge content. It transforms the related semantic elements into specific filtering conditions or weighting factors. For example, if the related semantic element contains instances, the search condition will tend to find content units marked as cases or containing specific examples; if the element emphasizes formula derivation, it will increase the weight of content units containing only textual derivation steps or accompanying derivation videos.

[0100] In step S53, based on the second search criteria, a secondary screening and matching process is performed on the candidate content unit set. The system traverses each content unit in the candidate set, evaluating its metadata, such as detailed tags, summary descriptions, and content fragments, to assess their compatibility with the second search criteria. This evaluation may be based on secondary keyword matching or semantic similarity calculation. Without describing the specific algorithms, it can be understood as matching or rule-based judgment based on semantic relationships, calculating a semantic relevance score for each candidate unit.

[0101] S54: Based on the semantic relevance score, select at least one content unit with the highest relevance from the candidate content unit set as the basic content unit. The system sets a relevance threshold and selects units with scores exceeding the threshold, or directly selects the top N units. These selected units are considered to be the knowledge materials that can most comprehensively and accurately answer the user's specific questions.

[0102] Furthermore, in step S6, determining the target modal format of the response is specifically as follows:

[0103] S61: Based on the target subject category, search for one or more candidate modal formats supported by the subject category from the preset subject-modal mapping table;

[0104] S62: Analyze the associated semantic elements to determine whether they contain implicit requirements for specific presentation forms, interaction methods, or logical display order;

[0105] S63: Combining the candidate modal formats and the implicit requirements, select or combine the candidate modal formats to generate the most suitable modal format as the target modal format.

[0106] In this embodiment, in step S61, one or more candidate modal formats supported by the target subject category are searched from a preset subject-modality mapping table. This mapping table defines the response formats typically applicable to different subject areas. For example, the mapping table might specify that mathematics is associated with step-by-step derivation and formula-chart combination formats; history is associated with timeline narrative and causal relationship diagram formats.

[0107] S62 analyzes the associated semantic elements to determine whether they contain implicit requirements for specific presentation formats, interaction methods, or logical display sequences. The system parses the associated semantic elements to find clues related to the presentation format. For example, user questions using phrases like "draw a diagram," "list steps," "compare in a table," or "present in chronological order" directly indicate format preferences. Even without direct indication, certain elements can hint at requirements, such as advantages and disadvantages suggesting a comparison format, or the development process suggesting a timeline or workflow format.

[0108] In step S63, combining the candidate modal formats and the implicit requirements, a most suitable modal format is selected or combined from the candidate modal formats to generate the target modal format. If a candidate modal format fully meets the implicit requirements, it is directly selected. If the implicit requirements require combining the characteristics of multiple formats, for example, requiring both step-by-step derivation and incorporating schematic diagrams, the system generates a composite modal format template according to predefined format combination rules. If none of the candidate formats are a perfect match, the closest format is selected, or the default format of the discipline is used.

[0109] Furthermore, the method also includes:

[0110] In the subject-modal mapping table, the candidate modal formats associated with science subjects include step-by-step derivation, formula and chart combination, and dynamic simulation demonstration; the candidate modal formats associated with humanities subjects include timeline narrative, comparative analysis table, and structured outline; and the candidate modal formats associated with arts subjects include work example display, style comparison atlas, and creative process decomposition.

[0111] The subject-modal mapping table:

[0112] Candidate modal formats related to science subjects such as mathematics, physics, and chemistry primarily serve logical deduction and precise expression. These include step-by-step derivation formats, which break down the solution process into sequential logical steps and present them in turn; formula-chart combination formats, which juxtapose core formulas with explanatory charts, coordinate graphs, or schematic diagrams; and dynamic simulation demonstration formats, which simulate physical processes, chemical changes, or geometric transformations through interactive or automatically playing animations.

[0113] Candidate modal formats related to humanities disciplines such as history, literature, and philosophy focus on narrative, analysis, and the organization of viewpoints. These include timeline narratives, which arrange events, works, or ideas in chronological order and connect them into a narrative line; comparative analysis tables, which use tables to compare the characteristics or strengths and weaknesses of different objects, viewpoints, and periods; and structured outlines, which summarize core content, arguments, or knowledge frameworks in the form of a well-organized list of key points or a mind map.

[0114] Candidate modal formats related to art disciplines such as fine arts and music focus on sensory experience and style analysis. These include: artwork example display format, which directly displays relevant paintings, musical score fragments, and audio / video samples with positioning descriptions; style comparison atlas format, which arranges images of works from different styles, schools, or periods side by side to highlight visual or auditory differences; and creation process breakdown format, which uses a series of pictures or segmented videos to gradually show the creation steps of an artwork or the components of a piece of music.

[0115] Furthermore, step S7 involves arranging and formatting the retrieved basic content units, including:

[0116] S71: Parse the content organization structure rules of the target modal format to determine the arrangement order, hierarchical relationship and logical connection method of the multiple basic content units;

[0117] S72: Parse the media presentation rules of the target modal format, and assign corresponding presentation controls or display templates to the text, image, audio or video content in the multiple basic content units;

[0118] S73: According to the content organization structure rules and media presentation format rules, the multiple basic content units and their metadata are encapsulated into response data packets that can be independently rendered or played.

[0119] In this embodiment, step S71 parses the content organization rules of the target modal format to determine the arrangement order, hierarchical relationship, and logical connection method of the at least one basic content unit. For example, if the format is a step-by-step derivation, the rules will require the content units to be arranged linearly according to logical cause and effect or chronological order, and each step may need to have a number and a title. If the format is a comparative analysis table format, the rules will require the content units to be assigned to the rows and columns of the table, and determine the table header and comparison dimensions.

[0120] In step S72, the system parses the media presentation rules of the target modal format and assigns corresponding presentation controls or display templates to the text, image, audio, or video content in the at least one basic content unit. The system determines the final appearance of each content unit based on its media type and format rules. Text may be placed in a specific text box style or dialog box; images may be resized and embedded within an icon frame; audio and video may have playback control buttons. The rules also define the layout relationships between different media elements, such as text wrapping around an image, or explanatory text below a chart.

[0121] In S73, according to the content organization structure rules and media presentation format rules, the at least one basic content unit and its metadata are encapsulated into a response data packet that can be independently rendered or played. The system generates a structured data object or file, which not only contains the original content but also detailed layout instructions, style definitions, and interaction logic descriptions. This data packet can be directly parsed by the front-end rendering engine and presented as a pre-designed multimodal interface.

[0122] Furthermore, the method also includes an interaction optimization step following step S8:

[0123] S9: After outputting the multimodal response content, receive feedback or follow-up questions from the user regarding the multimodal response content;

[0124] S10: Analyze the feedback or follow-up questions to determine whether they constitute a deepening or correction of the core question's intent or a supplement to the related semantic elements;

[0125] S11: If it is determined to be a deepening, correction or supplement, the feedback or follow-up question will be used as a new related semantic element. Combined with the original core question intent and the target subject category, a new round of semantic parsing, knowledge retrieval and content generation process starting from step S4 will be triggered to generate updated multimodal response content.

[0126] S12: Output the updated multimodal response content.

[0127] In this embodiment, after outputting the multimodal response content in step S9, the system receives feedback or follow-up questions from the user regarding the multimodal response content. The user may provide further interactive information through text input, clicking specific options on the interface such as "I don't understand here," "Please explain in another way," or marking a part of the answer. The system continuously listens for and captures these interactive events and their content.

[0128] In S10, the system analyzes the feedback or follow-up questions to determine whether they constitute a deepening, modification, or supplement to the core question's intent. The system compares the new user input with the core question's intent from the previous round. If the new input requests deeper, more specific details within the original intent, such as "How did the first step come about?", it is considered a deepening. If the new input changes some conditions of the original question or the direction of the inquiry—for example, "What if we don't consider friction?"—it is considered a modification. If the new input adds additional background information or limitations, such as "I'm a primary school student, please explain it more simply," it is considered a supplement.

[0129] If step S11 determines that the feedback or follow-up question is a deepening, correction, or supplement, then the feedback or follow-up question is treated as a new semantic element. Combined with the original core question intent and the target subject category, this triggers a new round of semantic parsing, knowledge retrieval, and content generation, starting from step S4. The system retains the original question's core intent and subject classification, but uses the new interactive content as the most important semantic element input. This means the system will re-parse the requirements based on an updated and more precise context, search for more matching content in the knowledge base, and may adjust the response modality format according to new requirements, such as choosing a simpler diagram or more colloquial text, thereby generating updated multimodal response content.

[0130] In S12, the updated multimodal response content is output. The newly generated response content is presented to the user, replacing or appending it to the previous answer area, thus achieving a continuous, context-aware interactive question-and-answer experience.

[0131] Furthermore, the method also includes:

[0132] The multimodal knowledge base is constructed as follows:

[0133] The knowledge content is divided according to subject categories;

[0134] Within each subject category, the knowledge content is categorized according to the content type of concepts, principles, cases, and exercises;

[0135] Each knowledge content unit is labeled with one or more semantic recognition strategy identifiers applicable to it and one or more recommended modal format identifiers.

[0136] In this embodiment, firstly, the knowledge content is categorized according to subject areas. This is the highest-level organization, ensuring that each knowledge unit has a clear subject affiliation. Secondly, within each subject area, the knowledge content is classified according to content types such as concepts, principles, cases, and exercises. This classification helps to quickly locate knowledge materials for different purposes; for example, concept-based units are used to explain basic concepts, while case-based units are used to illustrate applications. Finally, each knowledge content unit is labeled with one or more semantic recognition strategy identifiers and one or more recommended modal format identifiers. These identifiers are stored as metadata along with the content unit. The semantic recognition strategy identifier indicates which subject-based strategy's retrieval criteria are most suitable for the unit's content; the recommended modal format identifier suggests which presentation format is most suitable for the unit's content when organizing answers, providing direct reference for matching in step S5 and format selection in step S6.

[0137] According to a second embodiment of the present invention, the present invention claims protection for a teaching question-answering system based on a multimodal knowledge base, comprising:

[0138] One or more processors;

[0139] A memory that stores one or more programs, which, when executed by one or more processors, enable the processors to implement the teaching question-and-answer method based on a multimodal knowledge base.

[0140] The following is a specific example:

[0141] This embodiment is an intelligent education question-and-answer system deployed on a server, which has a built-in multimodal knowledge base, subject feature base, subject-modal mapping table, etc.

[0142] A learner interacts with the system through a terminal device such as a computer.

[0143] The user first enters a physics question: "System, please help me understand Newton's second law, preferably with examples of practical applications."

[0144] The system's input interface captured the complete text string entered by the user: "System, please help me understand Newton's second law, preferably with practical application examples."

[0145] The system extracts textual features from the request. It identifies the technical term "Newton's Second Law," a clearly defined core concept in physics. Additionally, the phrases "understanding" and "practical application examples" fall under the category of common teaching intent vocabulary.

[0146] The extracted features were matched with a subject-specific feature database. The database contains keywords related to physics, including Newton's second law, acceleration, force, and mass. Furthermore, the expression patterns of "understanding...laws and their connections...application examples" closely match typical question patterns for conceptual understanding and principle application in physics.

[0147] The matching score for physics was calculated to be significantly higher than that for other subjects such as mathematics, chemistry, and history. Therefore, the system determined physics as the target subject category.

[0148] The semantic recognition strategy is invoked: Based on the target subject of physics, the system invokes a question-and-answer semantic recognition strategy customized for physics from the subject feature library. The rules of this strategy are set as follows: first, identify the name of the physical law / principle; second, extract the physical quantity related to the principle; and finally, pay attention to the different emphases of the question on qualitative interpretation, quantitative calculation, or example application.

[0149] Semantic analysis:

[0150] Sentences are analyzed using semantic parsing rules from the field of physics. Dependency analysis shows that understanding is the core action, Newton's second law is its direct object, and connecting it to practical applications is the adverbial of purpose.

[0151] Based on the priority of physical strategies, the following was identified: the core action subject is the user, the core action object is Newton's second law, the core constraint is that the problem is empty and no specific conditions are specified, such as being in a vacuum, and the problem-solving goal is to understand and qualitatively grasp the concept while also connecting it to practical applications.

[0152] Map the above elements to a structured representation of the core problem intent: {Action: Explanation and understanding, Object: Newton's second law, Additional requirements: Attached application examples}.

[0153] Extract relevant semantic elements. Here, it is best to explicitly specify the relevant semantic elements as an additional requirement, which means that the answer should include application cases of the principle in real-world scenarios.

[0154] Knowledge base retrieval and matching:

[0155] Based on the core question intent {explanation and understanding of Newton's Second Law}, the first search criteria are constructed: subject tag = physics, content type tag = concept or principle, and subject keyword containing Newton's Second Law. The system retrieves multiple basic content units, such as the textual description of Newton's Second Law, the formula of Newton's Second Law (F=ma), and the physical meaning of Newton's Second Law, forming a candidate set.

[0156] Based on the associated semantic elements and application examples, a second search condition is constructed: the content should contain or be strongly related to application cases, examples, or practical problems.

[0157] A second screening was conducted on the candidate set. System evaluation revealed that the textual description unit of Newton's Second Law mainly described the definition, with moderate relevance; the formula unit focused on quantification, with moderate relevance; and the physical meaning unit involved qualitative interpretation, with moderate relevance. Simultaneously, the system expanded its search scope and discovered a unit with the content type of "case study": the "Analysis of Newton's Second Law in Car Starting and Braking" unit contained both textual descriptions and diagrams.

[0158] After calculating the semantic relevance, the system selects the physical meaning of Newton's second law as the unit with high relevance to understanding and the case analysis of Newton's second law in car starting and braking as the basic content unit with high relevance to application examples.

[0159] Determine the modal format:

[0160] Based on the target discipline of physics, the discipline-modal mapping table is queried to obtain candidate modal formats, including principle explanation combined with example demonstration and formula derivation demonstration.

[0161] The analysis of related semantic elements, along with application examples, implicitly requires that the answer should combine the explanation of principles with the demonstration of practical cases.

[0162] Considering both the candidate formats and implicit requirements, a format combining principle explanations with illustrative examples best fits the bill. This format requires organizing principle explanations and specific cases in a juxtaposed or interwoven manner, and using visual elements to aid in the explanation.

[0163] Content arrangement and packaging:

[0164] The system employs a content organization structure that combines explanations of principles with illustrative examples. The rules require first presenting a qualitative description of the core principles, then introducing a typical example, and mapping the key points of the principles to specific aspects of the example. The system determines the order accordingly: first, the core content of the physical meaning unit is placed, then the content of the example analysis unit is inserted, with logical connectors established between the two.

[0165] The system parses the media presentation rules for this format. The rules require the principle section to be presented as a highlighted text box, while the example section should be accompanied by a diagram or animated thumbnail. The system assigns a principle summary text box template to the text in the physical meaning unit and an embedded image display box to the diagrams in the example analysis unit.

[0166] According to the above rules, the text and image resources of the two content units, along with the layout instructions and style information, are encapsulated into a complete response data packet.

[0167] The output response sends the encapsulated data packet to the user terminal. After the user interface is rendered, it displays a content area containing two parts: the first part is a text box with a background color that clearly explains the physical meaning of Newton's second law, that force is the cause of changes in the state of motion of an object; the second part is a case study area, with the left side describing the process of a car accelerating when the power is greater than the resistance and braking when friction provides the reverse force, and the right side accompanied by a schematic diagram of the force analysis of a car.

[0168] Differences in modal formats across different disciplines: Subsequently, the user attempted to ask questions from different disciplines, and the system demonstrated the differences in modal format adaptation.

[0169] Historical question: Briefly describe the main stages and characteristics of the French Revolution.

[0170] The discipline is identified as history. According to the mapping table, the discipline of history is associated with a timeline narrative or a structured outline.

[0171] After semantic parsing, the associated semantic elements include a summary, stages, and features. The system's structured outline, which clearly lists each stage and its features, better meets the requirements for summary and feature comparison than a pure timeline.

[0172] The final answer is presented in a hierarchical list format: I. XX stage time, characteristics: 1...2...; II. YY stage...

[0173] Artistic Question: Compare the differences in color usage between Impressionist and Classical paintings.

[0174] The subject is identified as art, and a style comparison chart of art subjects is generated based on the mapping table.

[0175] The core of semantic parsing is to compare the differences between two styles in the use of color in a specific dimension. The system directly selects a style comparison atlas format.

[0176] The final result generates a two-column comparison table, with Impressionism on the left and Classicism on the right. Each row compares different color characteristics, such as light source color and inherent color, and embeds thumbnails of representative paintings from both schools next to each characteristic description for intuitive comparison.

[0177] The user followed up on the previous answer about Newton's second law by asking: "You just said that when a car accelerates, the power is greater than the resistance. What would be the difference if it were on a very smooth ice surface?"

[0178] Receive feedback / follow-up questions: The system captures this follow-up question.

[0179] Analysis and Judgment: This follow-up question is analyzed systematically. It doesn't pose a completely new question, but rather modifies the core constraint of the original core question—intending to understand Newton's second law and the existing example of car acceleration—by changing the ordinary road surface to a very smooth ice surface, i.e., with minimal friction. This attempts to deepen the understanding of the application boundaries and effects of the principle. Therefore, it is judged as a modification and deepening of the core constraint.

[0180] Trigger a new round of processing:

[0181] The system retains the original core problem intent {explanation and understanding, Newton's second law} and the target discipline physics.

[0182] The core of treating new follow-up questions as new semantic elements is to analyze the application results of the same principle on an ideal smooth surface with minimal friction.

[0183] This triggers a new process starting from S4. After semantic parsing S4, the new core intent structure remains unchanged, but the associated semantic elements are updated.

[0184] During knowledge retrieval S5, the second search condition becomes finding content involving low friction, ideal smooth surfaces, limiting cases, or changes in assumed conditions. The system may retrieve object motion analysis units on ideal smooth surfaces or parameter logic for dynamically adjusting existing instance units.

[0185] When determining the modal format S6, since the problem becomes a comparative analysis of the same principle under different conditions, the system may tend to enhance the comparative presentation based on the original format, for example, by adopting a sub-format that keeps the principle unchanged, compares the conditions, and describes the results separately.

[0186] After content arrangement and generation of S7 and S8, the system outputs an updated response. The new answer may be located below the original answer case area, with a new section for analyzing changes in conditions. This section uses text and modified diagrams, such as removing the friction markers between the tires and the ground, to explain that on a smooth ice surface, due to the greatly reduced resistance, the same force will produce greater acceleration, making the car start faster and also more difficult to brake. This deepens the understanding of how force changes the state of motion and the concept of net external force.

[0187] Knowledge base construction:

[0188] The intelligent educational question-and-answer system in this embodiment constructs its multimodal knowledge base as follows:

[0189] All knowledge content is first divided into major disciplines such as physics, history, and art.

[0190] Within the discipline of physics, the knowledge content is further categorized by type. For example, the textual description of Newton's second law is categorized as a concept, the formula for Newton's second law is categorized as a principle, the case study of car starting and braking is categorized as a case study, and practice questions about Newton's second law are categorized as exercises.

[0191] Each content unit is marked with rich metadata. For example, the metadata for the "Car Starting and Braking Case Analysis" unit, in addition to Subject = Physics and Type = Case, also includes:

[0192] Applicable semantic recognition strategy identifier: Phys_Strategy_1 This is an internal identifier that points to a principle- and application-oriented recognition strategy in the physics discipline.

[0193] Recommended modal format identifiers: Format_PrincipleWithExample (principle-based with examples) and Format_ComparativeAnalysis (comparative analysis). This indicates that the content of this unit is suitable for both explaining principles with examples and comparing different cases.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0195] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0196] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A teaching question-answering method based on a multimodal knowledge base, characterized in that, Includes the following steps: S1: Receive the user's original question-and-answer request; S2: Identify the subject category of the original question-and-answer request and determine its target subject category; S3: Based on the target subject category, call the corresponding question-and-answer semantic recognition strategy from the preset subject feature library. The question-and-answer semantic recognition strategy defines semantic parsing rules and key element extraction priorities that match the knowledge expression characteristics of the target subject category. S4: Using the invoked question-and-answer semantic recognition strategy, perform semantic parsing on the original question-and-answer request to identify the core question intent and multiple related semantic elements; S5: Based on the core question intent and associated semantic elements, perform retrieval and matching in a pre-built multimodal knowledge base to obtain multiple basic content units associated with the core question intent. The content units stored in the multimodal knowledge base are marked with content type tags and subject category tags. S6: Determine the target modal format of the response based on the target subject category and the associated semantic elements; wherein, the target modal format defines the combination rules of content organization structure and media presentation form; S7: Based on the target modal format, the retrieved multiple basic content units are arranged and formatted to generate multimodal response content that meets the requirements of the target modal format; S8: Output the multimodal response content to respond to the user's original question-and-answer request; Step S4 employs a question-and-answer semantic recognition strategy to perform semantic parsing on the original question-and-answer request, including: S41: Perform syntactic structure analysis and dependency relation analysis on the original question-answering request according to the semantic parsing rules defined in the question-answering semantic recognition strategy; S42: Based on the results of the syntactic structure analysis and dependency relation analysis, extract priorities according to the key elements defined in the question-answering semantic recognition strategy, and identify and label the core action subject, core action object, core limiting conditions and problem-solving objectives in the original question-answering request; S43: Map the identified core action subject, core action object, core limiting conditions, and problem-solving objective into a structured representation of the core problem intent; S44: Extract auxiliary descriptive information, contextual indicators, or comparison relation words from the original question-and-answer request, excluding the core elements, as the associated semantic elements.

2. The method according to claim 1, characterized in that, The process of identifying subject categories in step S2 includes: S21: Extract text features and / or symbol features from the original question-and-answer request; S22: Match the text features and / or symbol features with the predefined keyword sets and concept patterns of each subject category in the subject feature library; S23: Determine the target subject category to which the original question and answer request belongs based on the subject category keyword set and concept pattern with the highest matching degree.

3. The method according to claim 1, characterized in that, Step S5 involves retrieval and matching within a pre-built multimodal knowledge base, including: S51: Based on the core question intent, construct the first search condition, and initially filter out a set of candidate content units in the multimodal knowledge base whose content type tags and subject category tags both match the core question intent; S52: Based on the associated semantic elements, construct a second search condition, the second search condition including requirements for the details, attributes or relationships indicated by the associated semantic elements; S53: Based on the second retrieval condition, perform secondary screening and matching in the candidate content unit set, and calculate the semantic relevance between each candidate content unit and the second retrieval condition; S54: Based on the semantic relevance, select multiple content units with the highest relevance from the candidate content unit set as the basic content units.

4. The method according to claim 1, characterized in that, In step S6, determining the target modal format of the response is specifically as follows: S61: Based on the target subject category, search for one or more candidate modal formats supported by the target subject category from the preset subject-modal mapping table; S62: Analyze the associated semantic elements to determine whether they contain implicit requirements for specific presentation forms, interaction methods, or logical display order; S63: Combining the candidate modal formats and the implicit requirements, select or combine the candidate modal formats to generate the most suitable modal format as the target modal format.

5. The method according to claim 4, characterized in that, Also includes: In the subject-modal mapping table, the candidate modal formats associated with science subjects include step-by-step derivation, formula-chart combination, and dynamic simulation demonstration. Candidate modal formats for humanities disciplines include timeline narrative, comparative analysis table format, and structured outline format; candidate modal formats for arts disciplines include work example display format, style comparison atlas format, and creative process breakdown format.

6. The method according to claim 1, characterized in that, Step S7 involves arranging and formatting the retrieved basic content units, including: S71: Parse the content organization structure rules of the target modal format to determine the arrangement order, hierarchical relationship and logical connection method of the multiple basic content units; S72: Parse the media presentation rules of the target modal format, and assign corresponding presentation controls or display templates to the text, image, audio or video content in the multiple basic content units; S73: According to the content organization structure rules and media presentation format rules, the multiple basic content units and their metadata are encapsulated into response data packets that can be independently rendered or played.

7. The method according to claim 1, characterized in that, The method further includes an interaction optimization step after step S8: S9: After outputting the multimodal response content, receive feedback or follow-up questions from the user regarding the multimodal response content; S10: Analyze the feedback or follow-up questions to determine whether they constitute a deepening or correction of the core question's intent or a supplement to the related semantic elements; S11: If it is determined to be a deepening, correction or supplement, the feedback or follow-up question will be used as a new related semantic element. Combined with the original core question intent and the target subject category, a new round of semantic parsing, knowledge retrieval and content generation process starting from step S4 will be triggered to generate updated multimodal response content. S12: Output the updated multimodal response content.

8. The method according to claim 1, characterized in that, Also includes: The multimodal knowledge base is constructed as follows: The knowledge content is divided according to subject categories; Within each subject category, the knowledge content is categorized according to the content type of concepts, principles, cases, and exercises; Each knowledge content unit is labeled with one or more semantic recognition strategy identifiers applicable to it and one or more recommended modal format identifiers.

9. A teaching question-and-answer system based on a multimodal knowledge base, characterized in that, include: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement a teaching question-answering method based on a multimodal knowledge base according to any one of claims 1 to 8.

Images