Image-text knowledge question answering method and system based on text single-mode large model
By converting files to Markdown format and using LaTeX to represent formulas, the problem of handling multimodal information in traditional knowledge question-answering systems is solved, enabling efficient generation of illustrated answers, reducing computational resource consumption, and improving user experience and system performance.
Patent Information
- Application Number
- CN202511379099.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-25
- Publication Date
- 2026-02-27
AI Technical Summary
Traditional knowledge-based question-answering systems struggle to efficiently process and display multimodal information such as images and formulas, consume enormous computing resources, and are highly dependent on large multimodal models.
The files are uniformly converted to Markdown format, formulas are represented using LaTeX format, and image references in Markdown documents are converted into image tags to build a plain text knowledge base. The results are then generated using a text-based single-modal large model, producing illustrated answers.
It improves the accuracy and richness of answers, reduces computational resource consumption, enhances system efficiency and scalability, and provides a more comprehensive user experience.
Smart Images
Figure CN121579620A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, in particular to a text and image knowledge question and answer method and system based on a text single-modal large model. BACKGROUND
[0002] With the rapid development of artificial intelligence technology, knowledge question and answer systems have been widely used in various fields of daily life such as intelligent assistants and search engines. The continuous progress of these technologies enables users to obtain information more conveniently. However, traditional knowledge question and answer systems mainly focus on the processing and retrieval of text information, and it is difficult to efficiently process and display multi-modal information such as images and formulas. Therefore, how to effectively integrate non-text information such as images and formulas into the question and answer system has become a core challenge to improve system functionality and user experience.
[0003] Currently, most knowledge question and answer systems rely on vector databases for information storage and retrieval. By converting text into vector representation, these systems achieve efficient similarity search. However, these systems usually focus on pure text data, and the processing capacity for non-text information such as images and formulas is relatively limited. To solve this problem, multi-modal technology usually converts images into text descriptions or vectors in the corresponding space for image retrieval, and formulas are represented by LaTeX code. Although these conversion methods have solved the problem of expressing non-text information to some extent, they still face the challenge of huge consumption of computing resources. The conversion of multi-modal information relies on complex multi-modal large models, which require a large amount of computing resources and powerful hardware support. SUMMARY
[0004] The technical task of the present application is to solve the above problems, and provide a text single-modal large model-based image and text knowledge question and answer method and system, which can optimize the knowledge question and answer process, generate high-quality answers containing images and formulas, and greatly improve the accuracy of intelligent question and answer and user interaction experience.
[0005] The technical solution adopted by the present application to solve its technical problems is:
[0006] A text single-modal large model-based image and text knowledge question and answer method, the implementation of the method includes:
[0007] File format conversion: uniformly convert files to Markdown format, use LaTeX format to represent formulas, and convert image references in Markdown documents to image tags;
[0008] Knowledge base construction: construct a knowledge base according to the converted Markdown file, support efficient retrieval and content matching;
[0009] Knowledge Q&A: Based on the retrieved knowledge, it generates answers through a text-based single-modal large model and supports the display of formulas (Latex format) and images, providing high-quality answers with both text and images.
[0010] This method constructs a plain text knowledge base by uniformly converting files to Markdown format, representing formulas using LaTeX format, and converting image references in Markdown documents into image tags. It relies solely on a text-based unimodal large model for learning and generation, and outputs the final result by rendering LaTeX formulas and image tags. This not only generates high-quality, visually appealing answers but also effectively reduces computational resource consumption, improving system efficiency and scalability.
[0011] Furthermore, the specific implementation of file conversion and formatting is as follows:
[0012] For user-uploaded non-Markdown format files (DOCX), the format is first converted using a suitable tool. For DOCX files, the Pandoc tool is used to automatically convert the paragraph content within the DOCX file. The converted document is in Markdown format, where:
[0013] Text section: Simply retain the original content and format it;
[0014] Formula section: The formulas in the original text were analyzed and converted using LaTeX format;
[0015] Image section: Image reference format using Markdown! [Image Description](Image Path) References images in the text; the system will generate the image file path and store it.
[0016] In the document conversion process, formulas are an important element that cannot be ignored. The system parses the formulas in the DOCX document and accurately converts them into the standard LaTeX format to ensure that the formulas are accurately expressed in the final document. This process involves not only basic symbol conversion, but also careful identification and handling of different forms of expression that may appear in the formulas, such as special formats such as subscripts and superscripts.
[0017] The formula conversion involves formatting the Markdown content automatically generated by the Pandoc tool, automatically identifying and adjusting any non-standard parts; replacing the superscripts and subscripts "T1" and "T2" with the standard LaTeX format "$T_{1}$" and "$T_{2}$"; and automatically removing unnecessary escape characters (such as "~" and "^").
[0018] Furthermore, formulas in Word:
[0019]
[0020] Transforming to:
[0021] $\left(x+a\right)^{n}=\sum_{k=0}^{n}{\left(\frac{n}{k}\right)x^{k}a^{n-k}}$.
[0022] 0}^{n}{\left(\frac{n}{k}\right)x^{k}a^{n-k}}$.
[0023] Further, the picture tag replacement and mapping are implemented as follows:
[0024] The system parses each picture in the document, automatically generates a unique picture tag for it, and replaces the original picture reference with the generated tag;
[0025] For picture references in Markdown format:
[0026] 
[0027] Will be replaced by:
[0028] [image]filename-Picture 1 Description[ / image]
[0029] Where "filename-Picture 1 Description" is the unique picture tag of the picture, which is generated by the picture's file name and description information; in this way, each picture can obtain a clear identification in the subsequent processing and generation process; and in the replacement process, the system will save the mapping relationship between the unique identifier of the picture and the storage path of the picture; this mechanism ensures that the picture tag in the document always matches its corresponding picture path;
[0030] Using [image] and [ / image] as picture tag identifiers helps to clearly define the scope of the picture tag. This approach not only helps to quickly identify and locate the picture tag, but also enables efficient searching of the picture's storage path. Through this efficient processing method, the system can ensure fast response and high-quality output when processing Markdown documents.
[0031] Further, the knowledge base construction is implemented as follows:
[0032] Document content segmentation: The content of the Markdown document is segmented to ensure that each part effectively represents the information in the document; this process divides large paragraphs or chapters into smaller parts to facilitate subsequent vectorization processing;
[0033] Content vectorization encoding: using appropriate vectorization techniques to encode the document content, generating a vector representation of each part of the content; the core of this step is to convert text information into high-dimensional vectors to capture semantics and context;
[0034] Storage to vector database: store the generated vector data into an efficient vector database for subsequent query and retrieval operations. This step ensures that relevant content can be quickly and accurately retrieved from the vector database, improving the practicality of the knowledge base.
[0035] Further, the knowledge question and answer includes:
[0036] Question processing and retrieval: when the user inputs a question, the question is first preprocessed to retrieve relevant content from the knowledge base; to improve retrieval efficiency and accuracy, the system analyzes the context and structure of the user's question, uses a large model to deeply understand the question, and optimizes the current question based on historical questions to ensure that the question in the multi-round dialogue can be accurately answered; the system retrieves the knowledge base based on the optimized question, queries and obtains relevant reference content;
[0037] General answer generation: if relevant content cannot be found in the knowledge base, reference knowledge will not be passed, and a general answer will be generated using a text single-modal large model; the answer is generated based on the model's understanding of the question and is independent of specific knowledge base content; although these answers do not contain specific knowledge base information, they can still provide reasonable answers that meet user needs;
[0038] Knowledge base-based answer generation: when the system can successfully retrieve relevant knowledge from the knowledge base, the generated answer will be more accurate and detailed; the system will integrate the retrieved reference content, including text, formulas, and picture labels, to generate a detailed answer using a text single-modal large model; at the same time, the system will determine through the large model whether the retrieved reference content can provide example pictures for the user's question, and provide relevant picture labels if possible.
[0039] Further, if the system can provide pictures, the model will generate picture-based answers by combining relevant picture labels with prompt text, and will attach the corresponding picture labels in the output; if it cannot provide pictures, it will only generate text answers based on knowledge.
[0040] The entire process works through multiple-step decision-making and the coordinated work of functional modules to ensure that the question is accurately answered and picture label support is provided in appropriate cases, thereby achieving efficient user question response and output capabilities combining text and pictures.
[0041] The present application also claims to protect a text single-modal large model-based picture-text knowledge question and answer system, comprising:
[0042] A file format conversion module is configured to uniformly convert files into Markdown format, use LaTeX format to represent formulas, and convert picture references in Markdown documents into picture tags.
[0043] A knowledge base construction module is configured to construct a knowledge base based on the converted Markdown file.
[0044] A knowledge question and answer module is configured to generate answers through a text single-modal large model based on the retrieved knowledge, support the display of formulas (in LaTeX format) and pictures, and provide high-quality picture-text answers.
[0045] The system can implement the above method.
[0046] The application also claims a picture-text knowledge question and answer device based on a text single-modal large model, comprising at least one memory and at least one processor.
[0047] The at least one memory is configured to store machine-readable programs.
[0048] The at least one processor is configured to call the machine-readable programs to implement the above method.
[0049] The application also claims a computer readable medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the above method.
[0050] Compared with the prior art, the picture-text knowledge question and answer method and system based on a text single-modal large model has the following beneficial effects:
[0051] The application converts knowledge base content into Markdown format and combines LaTeX formulas and picture tags to improve the accuracy and richness of the answers generated by the system. When the system successfully retrieves relevant knowledge, the generated answers will be more detailed and can provide accurate answers by combining text, formulas, and picture tags. The text single-modal large model integrates these reference contents to determine whether to provide example pictures and provide picture tag support when appropriate. This combination of text and pictures allows users to obtain more comprehensive answers, especially in the explanation of complex problems such as mathematical theorems, which can combine formulas and pictures to convey information more intuitively.
[0052] In addition, the application relies on a pure text model to generate answers, reducing the consumption of computing resources and improving efficiency and scalability. This optimized process not only improves user experience but also reduces computational overhead while ensuring accuracy and depth. Overall, the application improves system performance and user satisfaction through precise knowledge retrieval and text-picture combination. BRIEF DESCRIPTION OF DRAWINGS
[0053] Figure 1 FIG. 1 is a flowchart of a method for text-based single-modal large model-based image-text knowledge question answering provided by an embodiment of the present application. DETAILED DESCRIPTION
[0054] The present application will be further described below in conjunction with the drawings and specific embodiments.
[0055] An embodiment of the present application provides a text-based single-modal large model-based image-text knowledge question answering method, which includes uniformly converting file content into Markdown format, and converting formulas into LaTeX format and picture references into picture tags; when generating an answer, the picture tags in the reference content are judged, and a text-image answer is generated in combination with the text. By relying only on a text-based single-modal large model to generate an answer containing formulas and picture tags, the consumption of computing resources is reduced, and the system efficiency and scalability are improved. By combining LaTeX formulas and picture tags, a detailed answer containing formulas and pictures is generated.
[0056] The implementation of the method includes:
[0057] File format conversion: uniformly converting files into Markdown format, using Latex format to represent formulas, and converting picture references in Markdown documents into picture tags,
[0058] Knowledge base construction: constructing a knowledge base according to the converted Markdown file, supporting efficient retrieval and content matching;
[0059] Knowledge question answering: based on the retrieved knowledge, generating an answer through a text-based single-modal large model, and supporting the display of formulas (Latex format) and pictures, providing a high-quality text-image answer.
[0060] Traditional rule-based and retrieval-based question answering methods are gradually transforming into intelligent question answering systems based on generative models. However, although generative question answering systems have strong natural language generation capabilities, they often perform poorly when faced with complex knowledge structures, image-text information, and formulas and other multi-modal data. In order to improve the performance of question answering systems in generating high-quality, text-image answers in a text knowledge base, the image-text knowledge question answering system based on the RAG (Retrieval-Augmented Generation) framework proposed by the present method can combine the advantages of traditional retrieval-based question answering and enhance the performance of text-based single-modal large models, providing high-quality text-image answers.
[0061] The specific implementation process is as follows:
[0062] 1. File processing and knowledge base construction
[0063] The construction of the knowledge base is a key step in the knowledge question answering system, which requires converting files of different formats into a unified format to ensure smooth retrieval and answer generation. This method mainly relies on Markdown format documents to store and present knowledge, and through preprocessing of document content, it provides effective support for subsequent knowledge question answering.
[0064] (1) File conversion and formatting:
[0065] For non-Markdown format files (DOCX) uploaded by users, first convert them to the appropriate format. For DOCX files, use the Pandoc tool to automatically convert the paragraph content in the DOCX file. The converted document uses Markdown format, which includes:
[0066] Text part: Directly retain the original content and format it.
[0067] Formula part: By analyzing the formulas in the original text, convert them to LaTeX format. For example, the formula in Word:
[0068]
[0069] will be converted to:
[0070] $\left(x+a\right)^{n}=\sum_{k=0}^{n}{\left(\frac{n}{k}\right)x^{k}a^{n-k}}$.
[0071]
[0072] Picture part: Reference the picture in the text by the Markdown picture reference format! [Picture description] (Picture path) to generate the path of the picture file and store it.
[0073] During the document conversion process, formulas are an important element that cannot be ignored. By analyzing the formulas in the DOCX document, accurately convert them to standard LaTeX format to ensure accurate expression of formulas in the final document. This process not only involves basic symbol conversion, but also requires careful identification and processing of different forms that may appear in the formula, such as subscripts and superscripts.
[0074] For example, when the system parses the subscript expression in the DOCX document, it may encounter some inconsistencies in the format. Specifically, the subscript content "T1" and "T2" in the DOCX file will be converted to "T~1~" and "T^2^", and the symbols "~" and "^" will be escaped to "\~" and "\^". This conversion method does not conform to the LaTeX standard and may cause the final rendered document to display the formula incorrectly.
[0075] To ensure that the conversion result meets the expectations, the system will format the Markdown content generated by the Pandoc tool automatically, automatically identify and adjust the parts that do not conform to the standard. For subscripts like "T1" and "T2", the standard LaTeX format "$T_{1}$" and "$T_{2}$" will be used to replace them, ensuring the correctness of the formula. At the same time, other unnecessary escape symbols (such as "\~" and "\^") are automatically removed to ensure the correct rendering of the formula.
[0076] (2) Picture tag replacement and mapping:
[0077] To ensure that the picture can be correctly output by the text single-modal large model and accurately replaced and rendered by the system, an efficient mechanism is designed to manage the relationship between the picture and its corresponding label. For pictures in Markdown documents, a unique label is automatically generated for each picture to ensure that each picture has a unique identifier in the entire knowledge base or document. These labels not only serve as reference identifiers for pictures, but also establish a clear mapping relationship with the actual storage path of the picture.
[0078] Specifically, each picture in the document is parsed, and a unique picture label is automatically generated for it, replacing the original picture reference with the generated label. For example, the picture reference in Markdown format:
[0079] ![Picture 1 Description] (Picture 1 Storage Path)
[0080] will be replaced by:
[0081] [image]filename-Picture 1 Description[ / image]
[0082] Where "filename-Picture 1 Description" is the unique picture label of the picture, which is generated by the picture's file name and description information. In this way, each picture can obtain a clear identification in the subsequent processing and generation process. And in the replacement process, the mapping relationship between the unique identifier of the picture and the picture path is saved. This mechanism ensures that the picture label in the document always matches its corresponding picture path.
[0083] Additionally, using [image] and [ / image] as identifiers for image tags helps to clearly define the scope of image tags. This approach not only aids in quickly identifying and locating image tags but also enables efficient searching of image storage paths. Through this efficient processing method, the system can ensure quick response and high-quality output when processing Markdown documents.
[0084] 2. Knowledge Base Construction
[0085] The processed Markdown files will be used to construct an efficient vector knowledge base. The specific steps are as follows:
[0086] Document content segmentation: The content of the Markdown document is segmented to ensure that each part effectively represents the information in the document. This process divides large paragraphs or chapters into smaller parts to facilitate subsequent vectorization processing.
[0087] Content vectorization encoding: Use appropriate vectorization techniques to encode the document content and generate vector representations of each part of the content. The core of this step is to convert text information into high-dimensional vectors to capture semantics and context.
[0088] Storage to vector database: Store the generated vector data into an efficient vector database for subsequent query and retrieval operations. This step ensures that relevant content can be quickly and accurately retrieved from the vector database, improving the practicality of the knowledge base.
[0089] 3. Knowledge Q&A
[0090] (1) Question processing and retrieval:
[0091] When the user inputs a question, the question is first preprocessed to retrieve relevant content from the knowledge base. To improve retrieval efficiency and accuracy, analyze the context and structure of the user's question, use a large model to understand the question deeply, and optimize the current question based on historical questions to ensure that questions in multiple rounds of dialogue can be accurately answered. Based on the optimized question, retrieve the knowledge base, query and obtain relevant reference content.
[0092] (2) General answer generation:
[0093] If the system cannot find relevant content in the knowledge base, it will not pass on reference knowledge and will directly use a text single-modal large model to generate a general answer. This type of answer is generated based on the model's understanding of the question and is independent of specific knowledge base content. Although these answers do not contain specialized knowledge base information, they can still provide reasonable answers that meet user needs.
[0094] (3) Knowledge base-based answer generation:
[0095] When the system can successfully retrieve relevant knowledge from the knowledge base, the generated answers will be more accurate and detailed. By integrating the retrieved reference content (including text, formulas, and image tags), a detailed answer is generated using a text single-modal large model. At the same time, the large model determines whether the retrieved reference content can provide example pictures for the user's question, and provides relevant image tags when possible.
[0096] If the system can provide pictures, the model will generate picture-based answers by combining relevant image tags with prompt text, and include the corresponding image tags in the output; if it cannot provide pictures, it will only generate text answers based on knowledge. Through multi-step decision-making and the collaborative work of functional modules, the entire process ensures accurate answers to questions and provides picture tag support in appropriate cases, thereby achieving efficient user question response and output capabilities with combined text and images.
[0097] For example, when a user asks for the proof of a certain theorem, the system will extract the proof process of the theorem from the knowledge base, which may include LaTeX-formatted formulas. The text single-modal large model will automatically include these LaTeX formulas when generating answers, ensuring that users receive accurate and in-depth answers.
[0098] This optimization process not only improves the accuracy of answers, but also enhances the ability to combine text and images, providing a more comprehensive and rich user experience.
[0099] The embodiments of the present application also provide a text single-modal large model-based image-text knowledge question-answering system, comprising:
[0100] A file format conversion module is used to convert files uniformly into Markdown format, represent formulas using Latex format, and convert image references in Markdown documents into image tags.
[0101] A knowledge base construction module is used to construct a knowledge base according to the converted Markdown files.
[0102] A knowledge question-answering module is used to generate answers based on retrieved knowledge through a text single-modal large model, support the display of formulas (Latex format) and images, and provide high-quality answers with combined text and images.
[0103] The system can implement the text-based single-modal large model-based image-text knowledge question answering method described in the above embodiments. The system first converts non- Markdown format documents (such as docx format) into Markdown format. During the conversion process, the system automatically converts the formulas in the document into Latex format and converts the pictures in the document into Markdown format picture references (![Picture Description] (Picture Path)). Subsequently, the system processes the Markdown format document, generates a unique label for each picture based on the picture description ([image] file name-picture description [ / image]), and establishes a mapping between the label and the corresponding picture path, and replaces the picture reference with the picture label. The processed Markdown file is used as the basis for the knowledge base, containing text, Latex formulas, and picture labels, facilitating subsequent retrieval. When a user asks a question, the system will retrieve relevant content in the knowledge base based on the question and use a text-based single-modal large model to generate an answer containing Latex formulas and picture labels. The system generates answers containing Latex formulas and picture labels, ensuring that the formulas are correctly rendered on the front end and the pictures are dynamically loaded through the mapping relationship with the labels, thereby achieving a picture-text presentation effect. The system optimizes the knowledge question answering process and significantly improves the accuracy of intelligent question answering and user interaction experience by generating high-quality answers containing text, images, and formulas.
[0104] The file format conversion module includes:
[0105] (1) File conversion and formatting:
[0106] For user-uploaded non- Markdown format files (DOCX), the system first performs format conversion using appropriate tools. For DOCX files, the system uses the Pandoc tool to automatically convert the paragraph content in the DOCX file. The converted document uses Markdown format, which includes:
[0107] Text part: Directly retain the original content and format.
[0108] Formula part: By analyzing the formulas in the original text, use LaTeX format for conversion. For example, the formula in Word:
[0109]
[0110] will be converted to:
[0111] $\left(x+a\right)^{n}=\sum_{k=
[0112] 0}^{n}{\left(\frac{n}{k}\right)x^{k}a^{n-k}}$.
[0113] Picture Part: Refer to the picture in the text by the Markdown picture reference format! [Picture Description] (Picture Path). The system will generate the path of the picture file and store it.
[0114] In the document conversion process, formulas are an important element that cannot be ignored. The system accurately converts the formulas in the DOCX document to standard LaTeX format by parsing them, ensuring the accurate expression of formulas in the final document. This process not only involves basic symbol conversion, but also requires careful identification and processing of different forms that may appear in the formula, such as subscripts and superscripts.
[0115] For example, when the system parses the subscript expression in the DOCX document, it may encounter some inconsistencies in the format. Specifically, the subscript content "T1" and "T2" in the DOCX file will be converted to "T~1~" and "T^2^", and symbols such as "~" and "^" will be escaped to "\~" and "\^ " in the automatic conversion process of the Pandoc tool. This conversion method does not conform to the LaTeX standard and may cause the final rendered document to fail to correctly display the formula.
[0116] In order to ensure that the conversion result meets the expectations, the system will format the Markdown content automatically generated by the Pandoc tool, automatically identify and adjust the parts that do not conform to the standard. For subscripts like "T1" and "T2", the system will replace them with LaTeX standard formats "$T_{1}$" and "$T_{2}$", ensuring the correctness of the formula. At the same time, for other unnecessary escape symbols (such as "\~" and "\^"), the system will automatically remove them to ensure the correct presentation of the formula.
[0117] (2) Picture tag replacement and mapping:
[0118] The system parses each picture in the document and automatically generates a unique picture tag for it, replacing the original picture reference with the generated tag. For example, the picture reference in Markdown format:
[0119] ! [Picture 1 Description] (Picture 1 Storage Path)
[0120] will be replaced by:
[0121] [image] file name - picture 1 description [ / image]
[0122] where "filename-picture1 description" is a unique picture tag for the picture, generated from the picture's filename and description information. In this way, each picture can be uniquely identified during subsequent processing and generation. During the replacement process, the system saves the mapping between the unique identifier of the picture and the picture path. This mechanism ensures that the picture tag in the document always matches its corresponding picture path.
[0123] In addition, the system uses [image] and [ / image] as identifiers for picture tags, helping to clearly define the scope of picture tags. This approach not only helps quickly identify and locate picture tags, but also efficiently finds the storage path of the picture. Through this efficient processing method, the system can ensure fast response and high-quality output when processing Markdown documents.
[0124] The knowledge base construction module includes:
[0125] Document content segmentation: The content of the Markdown document is segmented to ensure that each part effectively represents the information in the document. This process divides large paragraphs or chapters into smaller parts to facilitate subsequent vectorization processing.
[0126] Content vectorization encoding: Use appropriate vectorization technology to encode the document content and generate vector representation of each part of the content. The core of this step is to convert text information into high-dimensional vectors to capture semantics and context.
[0127] Storage to vector database: Store the generated vector data into an efficient vector database for subsequent query and retrieval operations. This step ensures that relevant content can be quickly and accurately retrieved from the vector database, improving the practicality of the knowledge base.
[0128] The knowledge question and answer module includes:
[0129] (1) Question processing and retrieval:
[0130] When the user inputs a question, the system first preprocesses the question to retrieve relevant content from the knowledge base. To improve retrieval efficiency and accuracy, the system analyzes the context and structure of the user's question, uses a large model to deeply understand the question, and optimizes the current question based on historical questions to ensure that the question in the multi-round dialogue can be accurately answered. The system retrieves the knowledge base based on the optimized question, queries and obtains relevant reference content.
[0131] (2) General answer generation:
[0132] If the system fails to find relevant content in the knowledge base, the system will not pass reference knowledge and directly use the text single-modal large model to generate general answers. These answers are generated based on the model's understanding of the question and are independent of specific knowledge base content. Although these answers do not contain specialized knowledge base information, they can still provide reasonable answers that meet user needs.
[0133] (3) Knowledge base-based answer generation:
[0134] When the system can successfully retrieve relevant knowledge from the knowledge base, the generated answers will be more accurate and detailed. The system will integrate the retrieved reference content (including text, formulas, and picture tags) and use the text single-modal large model to generate detailed answers. At the same time, the system will determine through the large model whether the retrieved reference content can provide example pictures for the user's question, and provide relevant picture tags if possible.
[0135] If the system can provide pictures, the model will generate picture-based answers by combining relevant picture tags with prompt text, and include the corresponding picture tags in the output; if it cannot provide pictures, it will only generate text answers based on knowledge. Through multi-step decision-making and the collaborative work of functional modules, the entire process ensures that questions are accurately answered and picture tags are provided in appropriate cases, thereby achieving efficient user question response and output capabilities that combine text and pictures.
[0136] The embodiment of the present application also provides a text single-modal large model-based image-text knowledge question answering device, comprising: at least one memory and at least one processor;
[0137] The at least one memory is used to store machine-readable programs;
[0138] The at least one processor is used to call the machine-readable programs to realize the text single-modal large model-based image-text knowledge question answering method described in the above embodiments.
[0139] The embodiment of the present application also provides a computer readable medium, which stores computer instructions, and the computer instructions realize the text single-modal large model-based image-text knowledge question answering method described in the above embodiments when executed by a processor. Specifically, a system or device equipped with a storage medium can be provided, and the storage medium stores software program code that realizes the functions of any one of the above embodiments, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.
[0140] In this case, the program code read from the storage medium itself can realize the functions of any one of the above embodiments, so the program code and the storage medium storing the program code constitute a part of the present application.
[0141] The storage medium for supplying the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.
[0142] Further, it should be understood that not only the program code read by the computer is executed, but also the operating system or the like operating on the computer is caused to perform part or all of the actual operation based on the instructions of the program code, thereby realizing the function of any one of the above-described embodiments.
[0143] Further, it should be understood that the program code read by the storage medium is written into the memory provided in the expansion board inserted into the computer or the memory provided in the expansion unit connected to the computer, and then the CPU or the like mounted on the expansion board or the expansion unit is caused to perform part or all of the actual operation based on the instructions of the program code, thereby realizing the function of any one of the above-described embodiments.
[0144] The present application has been described in detail above by the accompanying drawings and preferred embodiments, however, the present application is not limited to these disclosed embodiments, and as those skilled in the art can know based on the above-described embodiments, more embodiments of the present application can be obtained by combining the code review means in the above-described different embodiments, and these embodiments are also within the protection scope of the present application.
Claims
1. A text-based single-modal large model for graph-text knowledge question answering, characterized in that, The implementation of this method includes: File format conversion: Convert files to Markdown format, use LaTeX format to represent formulas, and convert image references in Markdown documents to image tags; Knowledge base construction: Build a knowledge base based on the converted Markdown file, supporting efficient retrieval and content matching; Knowledge Q&A: Based on the retrieved knowledge, answers are generated through a text-based single-modal large model, and formulas and images are supported for display, providing high-quality, illustrated answers.
2. The graph-text knowledge question-answering method based on a text-based single-modal large model according to claim 1, characterized in that, The specific implementation of file conversion and formatting is as follows: For user-uploaded non-Markdown format files, the format is first converted using a suitable tool; for DOCX files, the Pandoc tool is used to automatically convert the paragraph content in the DOCX file, and the converted document is in Markdown format, where: Text section: Simply retain the original content and format it; Formula section: The formulas in the original text were analyzed and converted using LaTeX format; Image section: Image reference format using Markdown! [Image Description](Image Path) References images in the text; the system will generate the image file path and store it. The formula conversion involves formatting the Markdown content automatically generated by the Pandoc tool, automatically identifying and adjusting any parts that do not conform to the standard; replacing the superscripts and subscripts "T1" and "T2" with the standard LaTeX format "$T_{1}$" and "$T_{2}$"; and automatically removing unnecessary escape characters.
3. A text-based single-modal large model-based graph-text knowledge question answering method according to claim 1 or 2, characterized in that, Formulas in Word: Transform into: $\left(x+a\right)^{n}=\sum_{k= 0}^{n}{\left(\frac{n}{k}\right)x^{k}a^{nk}}$.
4. The graph-text knowledge question-answering method based on a text-based single-modal large model according to claim 1, characterized in that, The specific implementation of image tag replacement and mapping is as follows: The system parses each image in the document, automatically generates a unique image tag for it, and replaces the original image reference with the generated tag; For image references in Markdown format: ! [Image 1 Description](Image 1 Storage Path) It will be replaced with: [image]Filename - Image 1 Description[ / image] Among them, "filename-image1 description" is the unique image tag of the image, which is generated by the image's filename and description information; during the replacement process, the system will save the mapping relationship between the image's unique identifier and the image path; Use [image] and [ / image] as identifiers for image tags to clearly define the scope of image tags.
5. The graph-text knowledge question-answering method based on a text-based single-modal large model according to claim 1, characterized in that, The construction of the knowledge base specifically includes: Document content segmentation: The content of the Markdown document is segmented to ensure that each part can effectively represent the information in the document; Content vectorization encoding: The document content is encoded using vectorization technology to generate a vector representation of each part of the content; Store to vector database: Store the generated vector data in a high-efficiency vector database for subsequent query and retrieval operations.
6. The graph-text knowledge question-answering method based on a text-based single-modal large model according to claim 1, characterized in that, The knowledge-based Q&A includes: Problem processing and retrieval: When a user enters a problem, the problem is first preprocessed to retrieve relevant content from the knowledge base; the context and structure of the user's problem are analyzed, a large model is used to deeply understand the problem, and the current problem is optimized based on historical problems to ensure that the problem in multi-turn dialogue can be accurately answered; based on the optimized problem retrieval knowledge base, relevant reference content is queried and obtained; General answer generation: If no relevant content is found in the knowledge base, no reference knowledge will be passed on, and a general answer will be generated directly using the text-based single-modal large model; the answer is generated based on the model's understanding of the question and is independent of the specific knowledge base content; Knowledge-based answer generation: When the system successfully retrieves relevant knowledge from the knowledge base, it integrates the retrieved reference content, including text, formulas, and image tags, and uses a text-based single-modal large model to generate a detailed answer. At the same time, the large model determines whether the retrieved reference content can provide example images for the user's question, and provides relevant image tags if possible.
7. The graph-text knowledge question-answering method based on a text-based single-modal large model according to claim 6, characterized in that, If the system can provide an image, the model will generate an image-based answer by combining the prompt text with relevant image tags, and include the corresponding image tags in the output; if no image can be provided, it will only generate a text-based answer based on knowledge.
8. A text-based single-modal large model for graph-text knowledge question answering, characterized in that, include: The file format conversion module is used to convert files to Markdown format, use LaTeX format to represent formulas, and convert image references in Markdown documents into image tags. The knowledge base building module is used to build a knowledge base based on the converted Markdown file. The knowledge-based question-answering module generates answers based on retrieved knowledge using a text-based single-modal large model, and supports the display of formulas and images, providing high-quality, richly illustrated answers. The system is capable of implementing the method described in any one of claims 1 to 7.
9. A text-based single-modal large model for graph-text knowledge question answering, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 7.
10. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, implement the method described in any one of claims 1 to 7.