CLIP-based bimodal retrieval textbook question and answer reasoning method and system
By leveraging the dual-modal retrieval and reasoning collaboration mechanism of the CLIP framework, the problems of information loss and cross-modal semantic discontinuity in multimodal textbook question-and-answer sessions are solved. This enables direct association and accurate retrieval of image and text features, generating logically consistent and traceable answers, which are suitable for primary and secondary school education scenarios and reduce deployment costs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-03-13
AI Technical Summary
Existing LLM question-answering systems suffer from several drawbacks in multimodal textbooks, including loss of key information due to modality switching, cross-modal semantic fragmentation, retrieval mechanisms that are not adapted to the needs of multimodal collaboration, low matching accuracy, lack of reasoning guidance, inability to mimic textbook logic and susceptibility to illusions, and the need for model fine-tuning to consume a large amount of computing resources and labeled data. These issues make it impossible to meet the requirements of knowledge accuracy, real-time performance and cost control in primary and secondary school education scenarios.
A dual-modal retrieval and reasoning collaboration mechanism based on the CLIP framework is adopted. The semantic features of the text and the visual features of the image are directly extracted through CLIP's text encoder and image encoder. A multimodal retrieval library adapted to TQA scenarios is constructed, the 'question-reasoning-answer' triple is extracted and labeled with cross-disciplinary tags, and the retrieval is carried out in combination with the cosine similarity algorithm. The reasoning guidance module organizes the context into a standardized format and generates answers that are consistent with the logic of the textbook.
It achieves a direct association between image visual features and text knowledge points, improves the accuracy of retrieval and matching, suppresses the generation of illusions, and generates answers with a traceable chain of textual and image evidence, reducing deployment costs and adapting to the flexibility and low-cost requirements of primary and secondary school teaching platforms.
Smart Images
Figure CN121660092A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and educational technology, specifically to a CLIP-based dual-modal retrieval method and system for question-answering reasoning in textbooks. Background Technology
[0002] In the field of education, Large Language Models (LLMs) have become the core technological foundation of question-answering systems. Mainstream optimization directions include Retrieval Augmentation (RAG) technology and model fine-tuning. RAG technology, by combining external retrieval mechanisms, compensates for the static nature of pre-trained knowledge in LLMs and is applied in text-based question-answering scenarios. Fine-tuning technology improves model adaptability by injecting domain-specific data. With the widespread use of multimodal teaching materials containing text, images, and experimental charts, educational scenarios urgently require question-answering systems to have multimodal information processing capabilities, demanding that the system achieve collaborative reasoning between text and image knowledge.
[0003] However, existing technologies have significant bottlenecks. Traditional RAG systems build retrieval and representation logic around text data. When directly applied to multimodal tasks, they struggle to connect the reasoning chain between visual and textual information, failing to achieve deep fusion of images and text and accurate retrieval. Model fine-tuning consumes substantial computational resources and labeled data, and the knowledge remains static after training, resulting in high deployment costs and failing to meet the application requirements of "low cost and high flexibility" in primary and secondary schools. These shortcomings lead to existing LLM systems facing problems such as prominent illusions, weak cross-modal reasoning, and high resource consumption in multimodal textbook question answering, failing to meet the teaching requirements for knowledge accuracy, real-time performance, and cost control. Therefore, there is an urgent need for efficient and intelligent question answering technology solutions adapted to this scenario.
[0004] RAG-based LLM question answering technology can effectively solve the above problems. Its core architecture is "text retrieval-generation", and the specific process is as follows: In the multimodal teaching material data preprocessing stage, the text is structured and the image is converted into a text description through the image description generation model, and then concatenated with the associated text to form mixed text data; In the retrieval library construction stage, the mixed text and pure text knowledge points are generated into vectors through the embedding model and stored in the database. When the user request contains an image, the image is first converted into text and then concatenated with the question to form a retrieval query, and the mixed text fragments are matched through a similarity algorithm; In the generation stage, the retrieval fragment and the user request are directly input into the LLM, and the answer is generated by relying on the LLM itself to integrate information. No reasoning logic association or multimodal semantic alignment mechanism is designed.
[0005] However, current RAG-based LLM question-answering technology still has significant technical problems: 1) Modality conversion leads to the loss of key information. Image-to-text conversion cannot fully preserve detailed information and spatial logic, causing cross-modal semantic gaps and making the retrieved knowledge disconnected from user needs; 2) The retrieval mechanism is not adapted to the needs of multimodal collaboration and has not established a direct association between image visual features and text knowledge points, resulting in low retrieval matching accuracy and difficulty in supporting accurate reasoning; 3) Reasoning guidance is lacking. The retrieved information is not processed with textbook-style logical structure. LLM relies on its own pre-trained knowledge to generate answers, which cannot guarantee consistency with textbook logic, is prone to creating illusions, and lacks a traceable chain of textual and image evidence.
[0006] In summary, existing textbook question-answering reasoning methods suffer from several problems: modality switching leads to the loss of key information; cross-modal semantic fragmentation exists; retrieval mechanisms are not adapted to the needs of multimodal collaboration; matching accuracy is low; reasoning guidance is lacking; they cannot mimic textbook logic and are prone to producing illusions. They cannot guarantee consistency with textbook logic, are prone to producing illusions, lack traceable textual and graphical evidence chains, struggle to achieve deep textual and graphical fusion and accurate retrieval, require significant computational resources and labeled data for model fine-tuning, the knowledge remains static after training, deployment costs are high, and they cannot meet the requirements of teaching for accuracy, real-time delivery, and cost control. Summary of the Invention
[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a CLIP-based dual-modal retrieval method and system for textbook question-answering. This method employs a dual-modal retrieval and reasoning collaboration mechanism adapted to TQA scenarios, directly linking the visual features of textbook images with textual knowledge points. It fully connects the reasoning chain of "image and text information → reasoning logic → answer," and, based on the structured knowledge of the textbook, guides LLM (Limited Language Management) to mimic the scientific reasoning logic of the textbook through a "question-reasoning step-answer" triplet. This suppresses illusions and ensures traceability of the reasoning process. This method does not rely on massive computing resources, can flexibly connect to existing teaching platforms in primary and secondary schools, and reduces school deployment costs. It effectively solves the problems of modality switching leading to loss of key information, cross-modal semantic gaps, retrieval mechanisms not adapting to multimodal collaboration needs, low matching accuracy, lack of reasoning guidance, inability to mimic textbook logic, and susceptibility to illusions inherent in RAG-based LLM question-answering technology. The system comprises a data preprocessing module, a retrieval library construction module, a bimodal retrieval module, a reasoning guidance module, and an answer generation module. Each module achieves hierarchical data interaction and functional dependencies through pre-defined data interfaces and calling protocols. The system directly extracts text semantic features and image visual features using CLIP's text encoder and image encoder, fully preserving text information, image details, and spatial logic, achieving a direct correlation between visual and text features. The bimodal retrieval library construction module extracts "question-reasoning-answer" triples and labels them with cross-disciplinary tags. The bimodal retrieval module calculates the similarity between the query and the unified vector representation of the triples using a cosine similarity algorithm, establishing a direct correlation between image visual features and text knowledge points, providing strong support for accurate reasoning. The method is simple, effective, and does not rely on massive computing resources. It can be flexibly integrated with existing teaching platforms in primary and secondary schools, reducing school deployment costs and demonstrating promising application prospects and commercial development value.
[0008] The specific technical solution for implementing this invention is: a CLIP-based dual-modal retrieval textbook question-and-answer reasoning method. Its characteristic is the adoption of a dual-modal retrieval and reasoning collaborative mechanism adapted to TQA scenarios, realizing the direct association between textbook image visual features and textual knowledge points, fully connecting the reasoning chain of "image and text information → reasoning logic → answer," and based on the reasoning guidance system of structured knowledge in textbooks, guiding LLM to imitate the scientific reasoning logic of textbooks through the "question-reasoning step-answer" triplet, suppressing illusions and achieving traceability of the reasoning process. This textbook question-and-answer reasoning method specifically includes the following steps:
[0009] Step 1: Data Preprocessing
[0010] We acquire K-12 multimodal textbook data, perform cross-disciplinary labeling on the multimodal textbook data, and extract "question-reasoning-answer" triples from the multimodal textbook data. The interdisciplinary labels correspond one-to-one with the triples; the problems in the triples Includes text or image questions, reasoning The answer includes the scientific reasoning process from the textbook and connects the corresponding textual knowledge points with key areas of the image. It includes standard answers that are consistent with the knowledge in the textbook.
[0011] Step 2: Building the search library
[0012] An adaptive bimodal unified embedding technique based on the CLIP framework generates a unified vector representation for each triplet. and will represent the unified vector After being associated with the corresponding interdisciplinary tags, they are stored in a vector database to form a multimodal retrieval library.
[0013] Step 3: Bimodal Search
[0014] Receive user question and answer requests The question and answer request The input format can be text input, image input, or a combination of text and image input, and it can respond to question-and-answer requests. Generate the corresponding query unified vector representation The search scope is expanded to triples of all disciplines based on interdisciplinary tags, and a unified vector representation of the query is calculated using the cosine similarity algorithm. Unified vector representation of triples in multimodal retrieval database Based on the similarity, the three most similar triplets were retrieved.
[0015] Step 4: Guided Structured Reasoning
[0016] The three retrieved triples were organized into a standardized context format. Each of them From the question ,reasoning ,Answer It is constructed by sequential splicing.
[0017] Step 5: Answer Generation
[0018] Question and answer request With standardization context The input is fed into the Qwen-7B large language model, which mimics the scientific reasoning logic in a standardized context to generate answers that are consistent with textbook knowledge and have a traceable reasoning process. .
[0019] In step 2, a unified vector representation is generated. The specific process is as follows:
[0020] Step 2.1: For triples containing only text, use CLIP's text encoder. Generate text embedding vectors As a unified vector representation ;
[0021] Step 2.2: For triples containing only images, use the CLIP image encoder. Generate image embedding vectors As a unified vector representation ;
[0022] Step 2.3: For triples containing both text and images, process them separately using CLIP's text encoder. and image encoder Generate text embedding vectors With image embedding vector Calculate the average of the two. As a unified vector representation .
[0023] Step 3 generates the corresponding query unified vector representation. The specific process is as follows:
[0024] Step 3.1: If the question-and-answer request is text-only, use CLIP's text encoder. Generate text embedding vectors As a unified vector representation for queries ;
[0025] Step 3.2: If the question-and-answer request is for image-only input, use the CLIP image encoder. Generate image embedding vectors As a unified vector representation for queries ;
[0026] Step 3.3: If the question-and-answer request is a text-image combination input, process it separately through CLIP's text encoder. and image encoder Generate text embedding vectors With image embedding vector Calculate the average of the two. As a unified vector representation for queries .
[0027] Each of the steps in step 4 The specific splicing method is as follows:
[0028]
[0029] in, Indicate the problem ,reasoning ,Answer The order in which they are assembled.
[0030] The specific method for expanding the search scope based on interdisciplinary tags in step 3 is as follows: based on question and answer requests... For the subject areas involved, the system matches triples in the multimodal retrieval library that are associated with the subject tag and predefined cross-subject tags, and then uses the cosine similarity algorithm to select the three triples with the highest similarity.
[0031] The specific method for cross-disciplinary tagging of multimodal textbook data in step 1 is as follows: First, the knowledge point tags of the textbook data are matched from the predefined K12 cross-disciplinary tag library by machine tagging. Then, the machine tagging results are manually verified and corrected. The predefined K12 cross-disciplinary tag library contains "Chemistry-Environmental Science", "Mathematics-Physics" and "Biology-Chemistry" tags that are suitable for primary and secondary school subjects, and the tags correspond one-to-one with the cross-disciplinary knowledge points of K12 textbooks.
[0032] A CLIP-based bimodal retrieval textbook question-answering reasoning system is characterized by comprising: a data preprocessing module, a retrieval database construction module, a bimodal retrieval module, a reasoning guidance module, and an answer generation module. Each module achieves hierarchical data interaction and functional dependency through preset data interfaces and calling protocols. The data preprocessing module transmits batch "question-reasoning-answer" triples labeled with cross-disciplinary tags to the retrieval database construction module via a structured data interface, serving as the basic data source for retrieval database construction. The retrieval database construction module grants the bimodal retrieval module read, write, and query permissions for the multimodal retrieval database through a vector database calling interface, providing full vector data support for retrieval operations. The bimodal retrieval module pushes the three most similar triples selected in real time to the reasoning guidance module through a result transmission interface. The reasoning guidance module outputs standardized context through a context splicing interface. and the user question-and-answer requests received by the bimodal retrieval module. Both are passed to the answer generation module through the model input interface as input data for the Qwen-7B large language model.
[0033] The data preprocessing module is used to acquire K12 multimodal textbook data, perform cross-disciplinary labeling on the multimodal textbook data, and extract the "question-reasoning-answer" triplet. Furthermore, the interdisciplinary labels correspond one-to-one with the triples.
[0034] The retrieval library construction module is used for adaptive bimodal unified embedding technology based on the CLIP framework to generate a unified vector representation for each triplet. and represent the unified vector. After being associated with corresponding interdisciplinary tags, they are stored in a vector database to form a multimodal retrieval library.
[0035] The dual-modal retrieval module is used to receive user question-and-answer requests. Generate the corresponding unified vector representation for the query. By expanding the search scope based on interdisciplinary tags and using the cosine similarity algorithm, the three triples with the highest similarity were retrieved.
[0036] The reasoning guidance module is used to organize the three retrieved triples into a standardized context format. .
[0037] The answer generation module is used to process question and answer requests. With standardization context Input into the Qwen-7B large language model to generate a response that meets the requirements. .
[0038] Compared with the prior art, the present invention has the following beneficial technical effects and significant technical progress:
[0039] 1) The adaptive bimodal unified embedding technology based on the CLIP framework of this invention directly extracts text semantic features and image visual features through CLIP's text encoder and image encoder. It can generate a unified vector representation without modality conversion, fully preserves text information and image details and spatial logic, realizes direct association between visual and text features, solves the problem of cross-modal semantic discontinuity, and enables precise matching of retrieval knowledge with user needs.
[0040] 2) The multimodal retrieval library construction module of this invention extracts the "question-reasoning-answer" triplet and labels it with interdisciplinary tags. The dual-modal retrieval module combines the interdisciplinary tags to expand the retrieval scope. The cosine similarity algorithm is used to calculate the similarity between the query and the unified vector representation of the triplet, establishes a direct association between image visual features and text knowledge points, improves the accuracy of retrieval matching, and provides strong support for accurate reasoning.
[0041] 3) The reasoning guidance module of this invention organizes the retrieved triples into a standardized context format. The Qwen-7B large language model receives this context and the user's question-and-answer request, and generates an answer by imitating the scientific reasoning logic of the textbook. This realizes the textbook-style logical structured processing of the retrieved information, suppresses the generation of illusions, and at the same time makes the answer have a traceable chain of textual and graphical evidence. Attached Figure Description
[0042] Figure 1 This is a flowchart of the CLIP-based dual-modal retrieval textbook question-answering reasoning method of the present invention;
[0043] Figure 2This is a schematic diagram of the CLIP-based dual-modal retrieval textbook question-answering reasoning system of the present invention. Detailed Implementation
[0044] See Figure 1 A CLIP-based bimodal retrieval method for textbook question-answering reasoning includes the following steps:
[0045] Step 1: Data Preprocessing
[0046] Acquire K12 multimodal teaching material data, perform cross-disciplinary labeling on the multimodal teaching material data, and extract "question-reasoning-answer" triples from the multimodal teaching material data. The interdisciplinary tags correspond one-to-one with the triples, and the questions in the triples... Includes text or image questions, reasoning The answer includes the scientific reasoning process from the textbook and connects the corresponding textual knowledge points with key areas of the image. It includes standard answers that are consistent with the knowledge in the textbook.
[0047] Step 2: Building the search library
[0048] An adaptive bimodal unified embedding technique based on the CLIP framework generates a unified vector representation for each triplet. The unified vector representation After being associated with the corresponding interdisciplinary tags, they are stored in a vector database to form a multimodal retrieval library.
[0049] Step 3: Bimodal Search
[0050] Receive user question and answer requests The input form of the question-and-answer request can be text input, image input, or a combination of text and image input, and a corresponding unified query vector representation is generated for the question-and-answer request. The search scope is expanded to triples of all disciplines based on interdisciplinary tags, and the unified vector representation of the query is calculated using the cosine similarity algorithm. Unified vector representation of triples in the multimodal retrieval library Based on the similarity, the three most similar triplets were retrieved.
[0051] Step 4: Guided Structured Reasoning
[0052] The three retrieved triples were organized into a standardized context format. Each of them From the question ,reasoning ,Answer It is constructed by sequential splicing.
[0053] Step 5: Answer Generation
[0054] The question and answer request With the standardized context The input is fed into the Qwen-7B large language model, which mimics the scientific reasoning logic in a standardized context to generate answers that are consistent with textbook knowledge and have a traceable reasoning process. .
[0055] See Figure 2 A CLIP-based bimodal retrieval textbook question-answering reasoning system includes: a data preprocessing module, a database construction module, a modal retrieval module, a reasoning guidance module, and an answer generation module. The data preprocessing module is used to acquire K12 multimodal textbook data, perform cross-disciplinary tagging on the multimodal textbook data, and extract "question-reasoning-answer" triples. Furthermore, the interdisciplinary tags correspond one-to-one with the triples; the retrieval library construction module is used to generate a unified vector representation for each triple based on the CLIP framework's adaptive bimodal unified embedding technology. and represent the unified vector. After being associated with corresponding interdisciplinary tags, the data is stored in a vector database to form a multimodal retrieval library; the bimodal retrieval module is used to receive user question-and-answer requests. Generate the corresponding unified vector representation for the query. The search scope is expanded based on interdisciplinary tags, and the three most similar triples are retrieved using a cosine similarity algorithm. The reasoning guidance module is used to organize the three retrieved triples into a standardized contextual format. The answer generation module is used to process question-and-answer requests. With standardization context Input into the Qwen-7B large language model to generate a response that meets the requirements. .
[0056] The technical solutions of the present invention will now be clearly and completely described with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0057] Example 1
[0058] See Figure 1This embodiment details the implementation process of the CLIP bimodal retrieval textbook question-answering reasoning method in a K12 education scenario, covering key steps such as data preprocessing, retrieval database construction, bimodal retrieval, structured reasoning guidance, and answer generation, to achieve accurate question answering and traceable reasoning process for multimodal textbooks. The specific steps are as follows:
[0059] Step 1: Data Preprocessing
[0060] Acquire multimodal teaching material data adapted to K12 education scenarios. The data types include text fragments, image materials, and text-image combination materials to ensure that the data covers knowledge of all subjects in the K12 stage.
[0061] Step 2: Interdisciplinary Tagging
[0062] The process of cross-disciplinary labeling of multimodal teaching material data is as follows:
[0063] Step 2-1: Use machine annotation to match knowledge point tags from the predefined K12 interdisciplinary tag library with the textbook data. The predefined K12 interdisciplinary tag library includes tags such as "Chemistry-Environmental Science", "Mathematics-Physics" and "Biology-Chemistry", and each tag corresponds one-to-one with the interdisciplinary knowledge points in the K12 textbook.
[0064] Step 2-2: Professionals verify and correct the machine annotation results, eliminating similar erroneous annotations and supplementing missing annotations to achieve accurate binding between cross-disciplinary tags and textbook data;
[0065] Steps 2-3: Extract structured "question-reasoning-answer" triples from multimodal textbook data labeled with interdisciplinary tags. And each triple is associated with a cross-disciplinary label. Among them, the question... Includes text or image questions, reasoning The answer includes the scientific reasoning process from the textbook and clearly connects the corresponding textual knowledge points with key areas of the image. It includes standard answers that are completely consistent with the knowledge in the textbook, forming a logical closed loop of "question → reasoning → answer".
[0066] Step 3: Build the search library
[0067] Step 3-1: After completing data preprocessing, construct the retrieval library and generate a unified vector representation for each "question-reasoning-answer" triple using the adaptive bimodal unified embedding technique based on the CLIP framework. If the triplet All data is in text format. Input the entire text content of the triple into CLIP's text encoder. ,pass Extracting semantic features from text to generate text embedding vectors , directly As a unified vector representation of this triple .
[0068] If the triplet For image problems, For image key region analysis, To obtain the text answer corresponding to the image, input the image content from the triplet into CLIP's image encoder. ,pass Extracting visual features from an image to generate an image embedding vector. ,Will As a unified vector representation of this triple .
[0069] If the triple contains both text and image, input the text content separately. Generate text embedding vectors Input image content Generate image embedding vectors The average of the two is calculated using the arithmetic mean. The average value is used as the unified vector representation of the triple. ;
[0070] Step 3-2: Represent each triple as a unified vector. The corresponding interdisciplinary tags are associated with each other and stored in a vector database in batches to form a K12 textbook multimodal retrieval library that supports multimodal retrieval.
[0071] Step 4: Bimodal Search
[0072] Step 4-1: Based on the user's question and answer request The CLIP bimodal embedding technique, consistent with the construction of the retrieval database, is used to generate a unified vector representation for the query. ,like The input format is only text input; please request the text input. Generate text embedding vectors As ;like The input format is only image input; the image request input will be sent. Generate image embedding vectors As ;like The input format is a text-image combination input, which is obtained through... , Generate text embedding vectors Image embedding vector Calculate the average value As ;
[0073] Step 4-2: According to The search covers subject areas and matches all triples in the multimodal search library that are associated with the subject tag and predefined interdisciplinary tags, breaking the search limitations of a single subject and ensuring that interdisciplinary related knowledge is included in the search scope.
[0074] Step 4-3: Calculate the query unified vector representation using the cosine similarity algorithm. The unified vector representation of the triples after the above screening The similarity is calculated by the following formula:
[0075] .
[0076] Step 4-4: Sort the similarity results from high to low, and retrieve the three triples with the highest similarity as the core basis for subsequent reasoning.
[0077] Step 5: Structured Reasoning Introduction
[0078] The three retrieved triples are organized according to a predefined standardized context format. Each of them The assembly format strictly follows the order of "question-reasoning-answer", specifically represented by the following formula: .
[0079] In the formula, "⊕" indicates that the problem in the triplet is... ,reasoning ,Answer Concatenate the text in sequence, ensuring each... The logic is coherent and the structure is consistent, providing a clear logical paradigm for subsequent model reasoning.
[0080] Step 6: Generate the answer
[0081] User's question and answer request With standardization context The inputs are fed into the Qwen-7B large language model, which learns standardized context. 3 in The scientific reasoning logic, such as "identifying key regions in an image → associating them with relevant textual knowledge points → deriving a conclusion," is used to generate answers by mimicking this logic. ,and It must simultaneously meet two conditions: being completely consistent with the knowledge in K12 textbooks and containing a traceable reasoning process.
[0082] The beneficial technical effects and significant technical advancements of the above-described embodiment are as follows:
[0083] 1) Regarding interdisciplinary tagging and triple extraction, existing RAG-based LLM question-answering technologies do not perform interdisciplinary tagging on textbook data, and the knowledge is fragmented. This embodiment ensures the accuracy of tags through interdisciplinary tagging of "machine tagging + manual verification," providing a basis for interdisciplinary retrieval. The structured extraction of "question-reasoning-answer" triples enables textbook knowledge to form a logical closed loop, avoiding knowledge fragmentation and ultimately achieving accurate association of interdisciplinary knowledge. This provides sufficient basis for interdisciplinary question retrieval and provides structured knowledge units for subsequent reasoning.
[0084] 2) Regarding CLIP's dual-modal unified embedding technology, existing technologies achieve multimodal processing through "image-to-text," resulting in the loss of image details and cross-modal semantic gaps. This embodiment directly extracts features of each modality through CLIP's text / image encoder without modality conversion. Furthermore, the unified vector representation keeps the image and text in the same vector space, fully preserving textual semantic information and image visual details. This enables direct association between visual features and textual features, solving the problem of cross-modal semantic gaps and laying the foundation for accurate retrieval.
[0085] 3) In terms of interdisciplinary retrieval and cosine similarity screening, the existing technology is limited to a single discipline and cannot meet the needs of interdisciplinary questions. This embodiment expands the retrieval scope based on interdisciplinary tags and combines the cosine similarity algorithm to accurately match associated triples, ensuring that interdisciplinary questions can retrieve enough related knowledge. The selected high similarity triples provide strong relevant evidence for reasoning and improve the accuracy of reasoning.
[0086] 4) Regarding structured reasoning guidance and answer generation, existing technologies do not perform structured processing on retrieved information. LLM relies on pre-trained knowledge to generate answers, which is prone to illusion and lacks traceability. This embodiment provides clear reasoning logic through standardized context, guides Qwen-7B to imitate the reasoning paradigm of textbooks, suppresses the illusion phenomenon of LLM, and enables the answer to have a traceable reasoning process, which meets the requirements of educational scenarios for the accuracy and interpretability of knowledge.
[0087] Example 2
[0088] See Figure 2This embodiment illustrates the module structure, functional implementation, and data interaction logic between modules of the CLIP bimodal retrieval textbook question-answering reasoning system, providing practical support for the aforementioned question-answering reasoning method and adapting to the intelligent question-answering application of multimodal textbooks in K12 education scenarios. The system in this embodiment includes a data preprocessing module, a retrieval library construction module, a bimodal retrieval module, a reasoning guidance module, and an answer generation module. Each module achieves hierarchical data interaction and functional dependency through preset data interfaces and calling protocols. The data preprocessing module transmits "question-reasoning-answer" triples labeled with cross-disciplinary tags in batches to the retrieval library construction module in a standardized structured data format through a structured data interface, serving as the basic data source for retrieval library construction. The retrieval library construction module grants the bimodal retrieval module read, write, and query permissions for the multimodal retrieval library through a vector database calling interface, allowing the bimodal retrieval module to read triples from the library via this interface. The dual-modal retrieval module performs a search operation based on the corresponding tags and the selected three most similar triples via a result transmission interface. The inference guidance module then outputs standardized context through a context splicing interface. and received by the dual-modal retrieval module Together, they are passed to the answer generation module through the model input interface as input data for the Qwen-7B large language model.
[0089] The data preprocessing module is used to acquire K12 multimodal textbook data, complete cross-disciplinary labeling and "question-reasoning-answer" triple extraction. Specifically, it acquires multimodal textbook data through a data acquisition interface, matches labels from a predefined K12 cross-disciplinary label library using a built-in machine labeling model, and then receives manual verification results through a human interaction submodule to correct the labels. At the same time, it automatically extracts triples according to the triple extraction rules and establishes a one-to-one correspondence between triples and cross-disciplinary labels.
[0090] The retrieval library construction module is used to generate triplet unified vector representations based on the CLIP framework's adaptive bimodal unified embedding technology, and to construct a multimodal retrieval library. Specifically, it integrates CLIP's text encoder. With image encoder Generate only text triples Image triple generation only Text-image combination triple generation Obtain a unified vector representation Then, through the vector database write interface, The corresponding interdisciplinary tags are associated with and stored in a vector database to form a multimodal retrieval library.
[0091] The dual-modal retrieval module receives user question-and-answer requests, generates a unified query vector representation, and retrieves the three most similar triples. Specifically, it provides a multimodal input interface and calls the CLIP encoder to generate... ,based on The system matches the subject area with the corresponding triplets of cross-disciplinary tags, calculates the similarity through the cosine similarity calculation submodule, and selects the top 3 triplets.
[0092] The reasoning guidance module is used to organize the three retrieved triples into a standardized context format. Specifically, it receives the Top 3 triples output by the bimodal retrieval module and, according to... The splicing rules generate a standardized context. .
[0093] The answer generation module is used to... and The input is fed into the Qwen-7B large language model, which generates a qualified answer, specifically received through the model input interface. and It calls the built-in Qwen-7B large language model to guide the model to imitate The reasoning logic within generates content that is consistent with textbook knowledge and includes a traceable reasoning process. The results are then fed back to the user through the output interface.
[0094] The beneficial technical effects and significant technical advancements of the above-described embodiment are as follows:
[0095] 1) In terms of module functionality, the existing system cannot adapt to multimodal textbook processing, interdisciplinary retrieval, and structured reasoning guidance. Each module of this system addresses the core requirements of data structuring, bimodal retrieval, reasoning guidance, and answer generation. Moreover, the functions are deeply adapted to the K12 textbook question-and-answer scenario, enabling efficient processing and accurate retrieval of multimodal textbook data, providing clear reasoning guidance for LLM, ensuring the accuracy and traceability of generated answers, and meeting the application needs of primary and secondary schools for "low cost and high flexibility".
[0096] 2) Regarding inter-module interaction, existing systems lack a unified interface for inter-module interaction, resulting in low data transmission efficiency and poor compatibility. This system achieves hierarchical interaction through preset interfaces and protocols, ensuring the standardization and efficiency of data transmission, enabling smooth data interaction between modules without data gaps or transmission delays, and improving the overall system response speed. At the same time, the unified interface design allows the system to flexibly connect to existing teaching platforms in primary and secondary schools, reducing deployment costs.
[0097] In summary, this CLIP-based bimodal retrieval method and system for textbook question-answering reasoning utilizes CLIP's adaptive bimodal unified embedding technology. Through CLIP's text encoder and image encoder, it directly extracts text semantic features and image visual features, generating a unified vector representation without modality transformation. This fully preserves text information, image details, and spatial logic, achieving a direct association between visual and text features. It solves the problem of cross-modal semantic gaps, ensuring precise matching between retrieved knowledge and user needs. Furthermore, the multimodal retrieval library construction module extracts "question-reasoning-answer" triples and labels them with cross-disciplinary tags. The bimodal retrieval module expands the retrieval scope by combining these cross-disciplinary tags. The cosine similarity algorithm calculates the similarity between the query and the unified vector representation of the triples, establishing a direct association between image visual features and text knowledge points, improving retrieval matching accuracy, and providing strong support for precise reasoning. Furthermore, the CLIP-based dual-modal retrieval textbook question-and-answer reasoning method and system organizes the retrieved triples into a standardized context format in the reasoning guidance module. The Qwen-7B large language model receives this context and the user's question-and-answer request, and generates an answer by imitating the scientific reasoning logic of the textbook. This achieves textbook-style logical structuring of the retrieved information, suppresses the generation of illusions, and at the same time ensures that the answer has a traceable chain of textual and graphical evidence.
[0098] All modules involved in this system are hardware system modules or functional modules combining computer software programs or protocols with hardware in existing technology. The computer software programs or protocols involved in these functional modules are technologies known to those skilled in the art and are not improvements to this system. The improvement of this system lies in the interaction or connection relationships between the modules, that is, in improving the overall structure of the system to solve the corresponding technical problems that this system aims to address.
[0099] It will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A CLIP-based bimodal retrieval method for textbook question-answering reasoning, characterized in that, The question-and-answer reasoning in this textbook includes the following steps: Step 1: Data Preprocessing The acquired K12 multimodal textbook data was labeled with cross-disciplinary tags, and "question-reasoning-answer" triples were extracted from the multimodal textbook data. ,in , , The interdisciplinary tags and triples One-to-one correspondence; the triplet Problems in Includes text or image questions; reasoning Includes the scientific reasoning process from the textbook and connects it with corresponding textual knowledge points and key areas of images; Answer Includes standard answers consistent with textbook knowledge; Step 2: Building the search library The adaptive bimodal unified embedding method of the CLIP framework is used to generate a unified vector representation for each triple. They are then associated with corresponding interdisciplinary tags and stored in a vector database to form a multimodal retrieval library; Step 3: Dual-modal retrieval User's question and answer request Generate the corresponding query unified vector representation Based on interdisciplinary tags, the search scope is expanded to triples of all disciplines, and a unified vector representation of the query is calculated using the cosine similarity algorithm. Unified vector representation of triples in multimodal retrieval database The similarity was used to obtain the three most similar triples, which were used in the question-answering request. The input format can be text input, image input, or a combination of text and image input; Step 4: Guided Structured Reasoning The three retrieved triples are organized into a standardized context represented by the following formula. : ； in, Examples included for the context, and each From the question ,reasoning and answer Composed of sequential splicing; Step 5: Answer Generation Question and answer request With standardization context Inputting the Qwen-7B large language model, it mimics the scientific reasoning logic in standardized contexts to generate answers that are consistent with textbook knowledge and have traceable reasoning processes. .
2. The CLIP-based dual-modal retrieval textbook question-answering reasoning method according to claim 1, characterized in that, Unified vector representation in step 2 The generation specifically includes: Step 2.1: For triples containing only text, use CLIP's text encoder. Generate text embedding vectors As a unified vector representation ; Step 2.2: For triples containing only images, use the CLIP image encoder. Generate image embedding vectors As a unified vector representation ; Step 2.3: For triples containing text and images, use CLIP's text encoder. and image encoder Generate text embedding vectors respectively and image embedding vector And calculate the average of the two. As a unified vector representation .
3. The CLIP-based bimodal retrieval textbook question-answering reasoning method according to claim 1, characterized in that, The query unified vector representation in step 3 The generation specifically includes: Step 3.1: If a question-and-answer request is made For text input, use CLIP's text encoder. The generated text embedding vector As a unified vector representation for queries ; Step 3.2: If a question-and-answer request is made For image input, use CLIP's image encoder. Generated image embedding vector As a unified vector representation for queries ; Step 3.3: If a question-and-answer request is made For text-image combined input, the CLIP text encoder will be used. and image encoder The generated text embedding vectors respectively With image embedding vector Calculate the average of the two. As a unified vector representation for queries .
4. The CLIP-based dual-modal retrieval textbook question-answering reasoning method according to claim 1, characterized in that, In step 4 The specific splicing method is expressed by the following formula: ； in, Indicate the problem ,reasoning ,Answer The order in which they are assembled.
5. The CLIP-based bimodal retrieval textbook question-answering reasoning method according to claim 1, characterized in that, The specific method for expanding the search scope based on interdisciplinary tags in step 3 is as follows: based on question and answer requests... For the subject areas involved, the system matches triples in the multimodal retrieval library that are associated with the subject tag and predefined cross-subject tags, and then uses the cosine similarity algorithm to select the three triples with the highest similarity.
6. The CLIP-based dual-modal retrieval textbook question-answering reasoning method according to claim 1, characterized in that, The specific method for cross-disciplinary tagging of multimodal textbook data in step 1 is as follows: the knowledge point tags of the textbook data are matched from the predefined K12 cross-disciplinary tag library by machine annotation, and then the machine annotation results are verified and corrected by humans. The predefined K12 cross-disciplinary tag library contains "Chemistry-Environmental Science", "Mathematics-Physics" and "Biology-Chemistry" tags that are suitable for primary and secondary school subjects, and the tags correspond one-to-one with the cross-disciplinary knowledge points of K12 textbooks.
7. A textbook question-answering reasoning system based on the CLIP dual-modal retrieval textbook question-answering reasoning method, characterized in that, The textbook question-and-answer reasoning system includes: a data preprocessing module, a retrieval database construction module, a bimodal retrieval module, a reasoning guidance module, and an answer generation module. Each module achieves hierarchical data interaction and functional dependencies through preset data interfaces and calling protocols. The data preprocessing module performs cross-disciplinary tagging on the acquired K12 multimodal textbook data and extracts the "question-reasoning-answer" triplet. Furthermore, the interdisciplinary tags correspond one-to-one with the triples; the retrieval library construction module is based on the CLIP framework's adaptive bimodal unified embedding technology, generating a unified vector representation for each triple. After being associated with corresponding interdisciplinary tags, the data is stored in a vector database to form a multimodal retrieval library; the bimodal retrieval module will process user question-and-answer requests. Generate the corresponding unified vector representation for the query. The search scope is expanded based on interdisciplinary tags, and the three most similar triples are retrieved using a cosine similarity algorithm. The reasoning guidance module then organizes the three retrieved triples into a standardized context represented by the following formula. : ； The answer generation module will handle the question and answer request. With standardization context Input the Qwen-7B large language model to generate a response that meets the requirements. .
8. The textbook question-answering reasoning system constructed based on the CLIP dual-modal retrieval textbook question-answering reasoning method according to claim 7, is characterized in that, The data preprocessing module transmits "question-reasoning-answer" triples labeled with cross-disciplinary tags in batches to the retrieval library construction module through a structured data interface, serving as the basic data source for retrieval library construction. The retrieval library construction module grants the bimodal retrieval module read, write, and query permissions for the multimodal retrieval library through a vector database call interface, providing full vector data support for retrieval operations. The bimodal retrieval module pushes the three most similar triples selected in real time to the reasoning guidance module through a result transmission interface. The reasoning guidance module outputs standardized context through a context splicing interface. and the user question-and-answer requests received by the bimodal retrieval module. The data is passed to the answer generation module through the model input interface as input data for the Qwen-7B large language model.