Fusion method based on TextRAG and VisRAG

Abstract

The invention relates to the technical field of retrieval enhancement generation, in particular to a fusion method based on TextRAG and Vi sRAG, which comprises the following steps of: segmenting and coding problems; document loading and imaging are carried out; enhancing the document information; encoding the document; calculating the similarity; performing Top-k text picture retrieval; performing multi-mode question and answer; and generating an answer reference. According to the method, the TextRAG and the Vi sRAG are fused, further filtering can be directly carried out by means of a ReRank method of the TextRAG so as to obtain the most relevant text reference, meanwhile, when a subsequent large model answers, a reference position can be added to trace an answer result so as to increase the error-tolerant rate of the answer, and in addition, by means of the Vi sRAG, for some tables which are easy to split in the TextRAG, the TextRAG and the Vi sRAG are combined, so that the error-tolerant rate of the answer is increased. According to the multi-modal large model, pictures which are difficult to be completely described by characters, even some mind maps, logic codes and other multi-modal elements can be input into the multi-modal large model in a complete information form, so that the integrity of input data is ensured, and the accuracy of an output result is further ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of retrieval enhancement generation, and in particular to a fusion method based on TextRAG and VisRAG. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a technology that combines information retrieval with text generation. It aims to enhance the generation ability of language models by introducing external knowledge sources. In application scenarios such as knowledge question answering, the RAG method has significant advantages. It can use the information in the knowledge base to provide the model with more accurate and rich answer basis, reducing the uncertainty and error rate of the model-generated content. However, the traditional RAG method has some limitations. It mainly relies on text information for retrieval and answering, and insufficiently utilizes knowledge containing multimodal information such as images. In the RAG system, TextRAG focuses on retrieval enhancement at the text level, while VisRAG is committed to introducing visual information by converting documents into images for retrieval and enhanced generation.

[0003] TextRAG extracts the top-k text content related to the question through text similarity calculation, and has a certain effectiveness in processing text information. It can quickly locate the text paragraphs in the knowledge base that are semantically similar to the question, and provide textual basis for subsequent answers. However, the limitation of TextRAG is that it relies only on text information, and cannot fully explore the information value of some content that requires visual assistance to understand (such as documents containing images, charts, etc.), which may lead to incomplete or inaccurate answers.

[0004] Vis aims to integrate visual information into the RAG framework. By treating a page of a document as a picture, it utilizes rich visual information to provide a richer source of information for knowledge answering. Its advantage is that it can use the visual features in the image to supplement the text information, so that the model can better understand complex knowledge content. It is especially suitable for scenarios involving image description and chart interpretation. However, when VisRAG is used alone, due to the lack of deep mining of text information, it may be insufficient in text-related semantic understanding and retrieval accuracy, and cannot fully realize its potential in multimodal knowledge question answering.

[0005] The application of large text-image models (e.g., Qwen2-72B-VL) in the RAG field provides a possible solution to the above problems. These models have the ability to process multimodal information of text and images, understand the semantics of text and images at the same time, and establish connections between the two. In RAG applications, large text-image models can receive text questions and related images as input and generate answers that integrate text and image information. However, there is still a lack of a complete solution on how to effectively integrate large text-image models with TextRAG and VisRAG to build an efficient and accurate RAG system. Summary of the invention

[0006] The technical problem to be solved by the present invention is: in order to solve the technical problem that the existing RAG method is insufficient in utilizing multimodal information, the present invention provides a fusion method based on TextRAG and VisRAG. By fusing TextRAG and VisRAG, effective collaboration of text and image information is achieved, which can improve the accuracy and richness of knowledge questions and answers. At the same time, it can also optimize the retrieval and generation processes, and improve the overall performance of the system to adapt to complex practical application scenarios.

[0007] The technical solution adopted by the present invention to solve the technical problem is: a fusion method based on TextRAG and VisRAG, comprising the following steps:

[0008] S1. Question segmentation and encoding. Question segmentation refers to: segmenting the question using a character-level segmentation method to obtain a character-level word array;

[0009] S2, document loading and image conversion, according to the different types of documents in the knowledge base, the corresponding loading method is used to convert the documents into text form;

[0010] S3, document information enhancement: Use text segmentation method to segment text documents into text blocks, then use LLMs to generate text block tables and hypothetical query questions, then concatenate the title, hypothetical query and text content, perform character-level segmentation again, and then remove stop words, so that each text block can get a phrase;

[0011] S4, document encoding, concatenate the text title, hypothetical query, and text content of each text block, segment the text block into words using character-level segmentation, and encode each word after removing stop words;

[0012] S5, similarity calculation, by calculating the similarity between the question encoding and all the text block encodings in the knowledge base, the top 10 text blocks most relevant to the question are screened out;

[0013] S6, Top-k text image retrieval;

[0014] S7, multimodal question answering;

[0015] S8. Answer reference generation: adding reference tags to LLMs’ answers to increase the accuracy of knowledge question answering results.

[0016] Therefore, by integrating TextRAG and VisRAG, we can directly use the ReRank method of TextRAG to perform further filtering to obtain the most relevant text citations. At the same time, when answering in the subsequent large model, we can also add the reference location to trace the answer results to increase the tolerance of the answer. In addition, with the help of VisRAG, some tables that are easy to be split in TextRAG, pictures that are difficult to fully describe with words, and even some multimodal elements such as mind maps and logic codes can be input into the multimodal large model in the form of complete information, thereby ensuring the integrity of the input data and the accuracy of the output results.

[0017] It is committed to breaking through traditional limitations and bringing more outstanding performance to knowledge question answering. Compared with traditional methods, this method is unique in improving accuracy. Its core lies in the use of character-based token encoding technology, which goes deep into the most basic component of text - the character level. In natural language processing, tokenization often decomposes text into words or phrases, but this method focuses on character-level token processing. This fine-grained approach provides a new perspective for understanding and processing text, and is of great importance to improving the accuracy of knowledge question answering systems.

[0018] Build a knowledge question-answering system based on character-level token encoding, which can significantly improve performance while maintaining high running speed without adding additional knowledge input;

[0019] In terms of integrating TextRAG and VisRAG, this approach fully leverages the advantages of both. In text retrieval, TextRAG extracts relevant text content through text similarity calculation to provide basic text information for subsequent processing, while VisRAG focuses on introducing visual information and associating text with visual elements such as images. Through reasonable process design, text information and visual information can work together to contribute to knowledge question and answer.

[0020] Furthermore, the S1 comprises the following steps:

[0021] S1-1, use jieba word segmenter to segment the question into individual words;

[0022] S1-2, based on a commonly used stop word, remove the stop word to obtain a word array Q;

[0023] S1-3. Use a deep learning text encoder to encode each word into a 768-dimensional numerical vector

[0024] Furthermore, in S1-2, the expression of the word array Q is:

[0025] Q=Q1Q2...Q n ;

[0026] In S1-3, the i-th word Q in the question after removing stop words i Numeric vector of The expression is:

[0027]

[0028] The matrix V consisting of the encodings of all words Q The expression is:

[0029]

[0030] Where: Q i Represents the i-th word in the question after removing stop words.

[0031] Furthermore, in S2, the document loading function provided by the open source tool langchain is used to support documents in doc, docx, pdf, markdown, csv, xslx, txt, HTML, and PPT formats;

[0032] Document imaging means converting each page of a document into an image and saving it to a document image database to facilitate subsequent multi-modal large-model question answering.

[0033] Document imaging is the operation of converting each page of a document in various formats into an image.

[0034] Further, the S3 comprises the following steps:

[0035] S3-1, preset the maximum length of a text block and the maximum overlapping length between adjacent text blocks, and set the key characters for segmentation to "\n", ".", ",",".", "\t";

[0036] S3-2. Call the RecursiveCharacterTextSplitter method of langchain to split each text block.

[0037] Further, the S4 comprises the following steps:

[0038] S4-1. Connect the text title, hypothetical query, and text content in the following order:

[0039] "Title: {text title}

[0040] Query: {hypothetical query}

[0041] Body: {text content}";

[0042] S4-2, using the jieba word segmenter to segment the concatenated content, and then using a common stop word list to remove stop words to obtain a word array D;

[0043] S4-3. Convert each word in the word array D into a 768-dimensional numerical vector using the m3e text encoding model All words form a numerical matrix V D ;

[0044] In S4-2, the expression of the word array D is:

[0045] D=D1D2...D m ;

[0046] Where: D j Represents the jth word in the document after removing stop words

[0047] In S4-3, the numerical vector The expression is:

[0048]

[0049] Numerical matrix V D The expression is:

[0050]

[0051] Further, the S5 comprises the following steps:

[0052] S5-1. For the problem encoding matrix V Q and the text block encoding matrix V D , calculate the MaxSim value of question Q and text block D as the text similarity between question Q and text block D;

[0053] S5-2, for all text blocks D1, D2, ..., D v , calculate the MaxSim value of each of them and question Q respectively to get the scores s1, s2, ..., s v ;

[0054] S5-3, for s i Sort by size and filter out the top 10 s i The corresponding text blocks D1, D2, ..., D 10 ;

[0055] Where: s j D j and the MaxSim score of question Q;

[0056] In S5-1, the calculation formula of MaxSim value is:

[0057]

[0058] Further, the S6 comprises the following steps:

[0059] S6-1. According to the top-10 text blocks extracted in S5, locate the corresponding document and the document image corresponding to the page number of the document through its attributes doc_id and page_num. Assume that the corresponding document images are P1, P2, ..., P 10 ;

[0060] S6-2, delete duplicate document images according to doc_id and page number page_num to obtain document images P1, P2, ..., P n ;

[0061] Where: n≤10.

[0062] Further, in S7, the document images P1, P2, ..., P corresponding to the question and the top 10 most relevant text blocks are displayed. n By constructing the input of VLMs, the output of VLMs is the answer to the given problem.

[0063] Further, the S8 comprises the following steps:

[0064] S8-1, using ".", ","!","?","?","\n" as the end symbols of the orange, segment the LLMs' answers at the sentence level;

[0065] S8-2. Use the Jieba word segmenter to segment each sentence, remove stop words, and then perform word-level encoding to obtain the data matrix V of each sentence S. S ;

[0066] S8-3, calculate the MaxSim value of sentence S and the 10 related text blocks obtained in S5. If these 10 related text blocks D1, D2, ..., D 10 If the maximum MaxSim value is greater than the preset threshold value 0.7, execute S8-4, otherwise, execute S8-5;

[0067] S8-4, it is considered that the sentence needs to be marked with a reference tag, and the reference tag is the sequence number of the largest text block of MaxSim in these 10 related text blocks;

[0068] S8-5, believes that the sentence only plays an excessive role and does not need a reference label;

[0069] Where: The data matrix V of each sentence S S The expression is:

[0070]

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] By integrating TextRAG and VisRAG, we can directly use TextRAG's ReRank method to do further filtering to obtain the most relevant text citations. At the same time, in the subsequent large-model answers, we can also add reference locations to trace the answer results to increase the tolerance of the answers. In addition, with the help of VisRAG, some tables that are easy to be split in TextRAG, pictures that are difficult to fully describe with words, and even some multimodal elements such as mind maps and logical codes can be input into the multimodal large model in the form of complete information, thereby ensuring the integrity of the input data and the accuracy of the output results. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0074] Figure 1 is a flow chart of the fusion method based on TextRAG and VisRAG of the present invention;

[0075] Figure 2 is a control block diagram of the fusion method based on TextRAG and VisRAG of the present invention;

[0076] Figure 3 A process diagram for text segmentation of the present invention;

[0077] Figure 4 It is a process diagram of similarity calculation of the present invention. DETAILED DESCRIPTION

[0078] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0079] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0080] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0081] like Figures 1 to 4 As shown, it is the best embodiment of the present invention. The fusion method based on TextRAG and VisRAG in this embodiment includes the following steps:

[0082] S1. Question segmentation and encoding. Question segmentation refers to: segmenting the question using a character-level segmentation method to obtain a character-level word array;

[0083] S2, document loading and image conversion, according to the different types of documents in the knowledge base, the corresponding loading method is used to convert the documents into text form;

[0084] S3, document information enhancement: Use text segmentation method to segment text documents into text blocks, then use LLMs to generate text block tables and hypothetical query questions, then concatenate the title, hypothetical query and text content, perform character-level segmentation again, and then remove stop words, so that each text block can get a phrase;

[0085] S4, document encoding, concatenate the text title, hypothetical query, and text content of each text block, segment the text block into words using character-level segmentation, and encode each word after removing stop words;

[0086] S5, similarity calculation, by calculating the similarity between the question encoding and all the text block encodings in the knowledge base, the top 10 text blocks most relevant to the question are screened out;

[0087] S6, Top-k text image retrieval;

[0088] S7, multimodal question answering;

[0089] S8. Generate answer references and add reference tags to the answers of LLMs to increase the accuracy of the knowledge question and answer results. Therefore, by integrating TextRAG and VisRAG, it is possible to directly use the ReRank method of TextRAG to perform further filtering to obtain the most relevant text references. At the same time, in the subsequent large model answers, it is also possible to add reference locations to trace the answer results to increase the tolerance of the answer. In addition, with the help of VisRAG, some tables that are easy to be split in TextRAG, pictures that are difficult to fully describe with words, and even some multimodal elements such as mind maps and logical codes can be input into the multimodal large model in the form of complete information, thereby ensuring the integrity of the input data and then ensuring the accuracy of the output results; it is committed to breaking through traditional limitations and bringing more outstanding performance to knowledge questions; compared with traditional methods, this method is unique in improving accuracy; its core lies in the use of character-level token encoding technology to go deep into the most basic constituent elements of the text - —Character level; In natural language processing, tokenization often decomposes text into words or phrases, but this method focuses on character-level token processing; This fine-grained approach provides a new perspective for understanding and processing text, and is of great importance to improving the accuracy of knowledge question-answering systems; Building a knowledge question-answering system based on character-level token encoding does not require additional knowledge input, but can significantly improve performance while maintaining high operating speed; In terms of integrating TextRAG and VisRAG, this method gives full play to the advantages of both. TextRAG extracts relevant text content through text similarity calculation in text retrieval to provide basic text information for subsequent processing, while VisRAG focuses on introducing visual information, associating text with visual elements such as images, and through reasonable process design, enables text information and visual information to work together to contribute to knowledge question-answering.

[0090] In other words, when faced with a question, the system first tokenizes the input question and the text block in the knowledge base at the character level. This process is like breaking down complex text into fine character units, making subsequent processing more accurate. In this way, the system can more accurately retrieve information that is highly relevant to the question in the knowledge base and provide users with more accurate answers. By integrating TextRAG and VisRAG, the multimodal information of text and images is fully utilized to improve the accuracy and richness of knowledge questions and answers. Compared with the traditional RAG method, it can better understand and answer questions involving image-related content, and provide users with more comprehensive and intuitive answers. The optimized system performs better in dealing with complex problems, improves overall performance and user experience, and promotes the development of RAG technology in the field of multimodal knowledge questions and answers.

[0091] In this embodiment, S1 includes the following steps:

[0092] S1-1, use jieba word segmenter to segment the question into individual words;

[0093] S1-2, based on a commonly used stop word, remove the stop word to obtain a word array Q;

[0094] S1-3. Use a deep learning text encoder (m3e) to encode each word into a 768-dimensional numerical vector (word embedding);

[0095] In S1-2, the expression of word array Q is:

[0096] Q=Q1Q2…Q n ;

[0097] In S1-3, the i-th word Q in the question after removing stop words i Numeric vector of The expression is:

[0098]

[0099] The matrix V consisting of the encodings of all words Q The expression is:

[0100]

[0101] Where: Q i Represents the i-th word in the question after removing stop words.

[0102] In this embodiment, in S2, the document loading function provided by the open source tool langchain is used to support documents in doc, docx, pdf, markdown, csv, xslx, txt, HTML, and PPT formats; document imaging refers to: converting each page of the document into an image and saving it in the document image database to facilitate the subsequent multimodal large model question and answer; document imaging is the operation of converting each page of a document in a variety of formats into an image. Specifically, there are many operations for document imaging, among which the use of Python programming language is more flexible and has a wide range of applications; the basic idea is to install the corresponding Python library for documents of different formats, read the document content through code, and use the image processing library to draw it into an image for saving (for example: processing PDF documents with the help of PyPDF2 to read the content, and using the Pillow library to generate images; Word documents (doc / docx) use python-docx to parse the text and then use Pillow to draw images; Markdown documents first use the markdown library to parse the text, and then use Pillow to complete the image drawing).

[0103] Table 1 Document type and corresponding function table

[0104] Document Type Function Name doc / docx UnstructuredWordDocumentLoader pdf PyPDFLoader markdown UnstructuredMarkdownLoader csv CSVLoader xx UnstructuredExcelLoader txt UnstructuredHTMLLoader HTML UnstructuredFileLoader PPT UnstructuredPowerPointLoader

[0105] Table 2: Python library table required to convert related formats into images

[0106] Document Type Python Libraries Used doc / docx python-docx,pillow pdf PyPDF2, pillow markdown markdown,pillow csv pandas,pillow xx paddas, pillow txt selenium,pillow HTML pillow PPT python-pptx、Pillow

[0107] In this embodiment, S3 includes the following steps:

[0108] S3-1, preset the maximum length of a text block and the maximum overlapping length between adjacent text blocks, and set the key characters for segmentation to "\n", ".", ",",".", "\t";

[0109] S3-2, call the RecursiveCharacterTextSplitter method of langchain to split each text block. Specifically, to ensure the traceability of information, each text block is set to the format of Table 3.

[0110] Table 3

[0111] Field Name illustrate doc_id The document's identity id, unique in the current knowledge base text The text of the text block doc_name The name of the document in which the text block is located doc_title The title of the text block, which needs to be generated in subsequent steps syn_question The hypothetical query problem of the text block requires subsequent steps to generate emb_matrix Numeric matrix, numeric matrix of text blocks page_num The page number in the document (doc_id) that the text block contains

[0112] Specifically, S3 generates the title of the text block with the help of LLMs. The specific method is as follows: an LLMs input is formed through the Prompt project, and the output is the required text block title;

[0113] The prompt template is as follows:

[0114] “I will give you an article below. Create a title for the article below.

[0115] This is the article content: <Document content>

[0116] NOTE: Provide only a title, no additional text. The answer format starts with 'Title:' followed by the question. Do not write an empty title. "

[0117] S3 needs to use LLMs to generate some hypothetical queries for the text block. The specific method is: construct an LLMs input through the prompt project, and the output is the required hypothetical query;

[0118] The prompt template is as follows:

[0119] “I’ll give you an article below. What search queries or questions are relevant to this article or that this article can answer?

[0120] Each query is on a separate line.

[0121] This is the article content: <Document content>

[0122] Provide only the user query, no additional text. Format each query as 'Query:' followed by the question. Do not write an empty query. "

[0123] In this embodiment, S4 includes the following steps:

[0124] S4-1. Connect the text title, hypothetical query, and text content in the following order:

[0125] "Title: {text title}

[0126] Query: {hypothetical query}

[0127] Body: {text content}";

[0128] S4-2, using the jieba word segmenter to segment the concatenated content, and then using a common stop word list to remove stop words to obtain a word array D;

[0129] S4-3. Convert each word in the word array D into a 768-dimensional numerical vector using the m3e text encoding model All words form a numerical matrix V D ;

[0130] In S4-2, the expression of the word array D is:

[0131] D=D1D2...D m ;

[0132] Where: D j Represents the jth word in the document after removing stop words

[0133] In S4-3, the numerical vector The expression is:

[0134]

[0135] Numerical matrix V D The expression is:

[0136]

[0137] In this embodiment, S5 includes the following steps:

[0138] S5-1. For the problem encoding matrix V Q and the text block encoding matrix V D , calculate the MaxSim value of question Q and text block D as the text similarity between question Q and text block D;

[0139] S5-2, for all text blocks D1, D2, ..., D in the knowledge base v , calculate the MaxSim value of each of them and question Q respectively to get the scores s1, s2, ..., s v ;

[0140] S5-3, for s i Sort by size and filter out the top 10 s i The corresponding text blocks D1, D2, ..., D 10 ;

[0141] Where: s j D j and the MaxSim score of question Q;

[0142] In S5-1, the calculation formula of MaxSim value is:

[0143]

[0144] In this embodiment, S6 includes the following steps:

[0145] S6-1. According to the top-10 text blocks extracted in S5, locate the corresponding document and the document image corresponding to the page number of the document through its attributes doc_id and page_num. Assume that the corresponding document images are P1, P2, ..., P 10 ;

[0146] S6-2, delete duplicate document images according to doc_id and page number page_num to obtain document images P1, P2, ..., P n ;

[0147] Where: n≤10.

[0148] In this embodiment, in S7, the document images P1, P2, ..., P3 corresponding to the question and the contents of the top 10 most relevant text blocks are displayed. n By constructing the input of VLMs, the output of VLMs is the answer to the given problem.

[0149] Specifically, the input template is as follows:

[0150]

[0151] In this embodiment, S8 includes the following steps:

[0152] S8-1, using ".", ","!","?","?","\n" as the end symbols of the orange, segment the LLMs' answers at the sentence level;

[0153] S8-2. Use the Jieba word segmenter to segment each sentence, remove stop words, and then perform word-level encoding to obtain the data matrix V of each sentence S. S ;

[0154] S8-3, calculate the MaxSim value of sentence S and the 10 related text blocks obtained in S5. If these 10 related text blocks D1, D2, ..., D 10 If the maximum MaxSim value is greater than the preset threshold value 0.7, execute S8-4, otherwise, execute S8-5;

[0155] S8-4, it is considered that the sentence needs to be marked with a reference tag, and the reference tag is the sequence number of the largest text block of MaxSim in these 10 related text blocks;

[0156] S8-5, believes that the sentence only plays an excessive role and does not need a reference label;

[0157] Where: The data matrix V of each sentence S S The expression is:

[0158]

[0159] In summary, by integrating TextRAG and VisRAG, the present invention can directly use the ReRank method of TextRAG for further filtering to obtain the most relevant text references. At the same time, in the subsequent large model answers, the reference location can be added to trace the answer results to increase the fault tolerance of the answer. In addition, with the help of VisRAG, some tables that are easy to be split in TextRAG, pictures that are difficult to fully describe with words, and even some multimodal elements such as mind maps and logical codes can be input into the multimodal large model in the form of complete information, thereby ensuring the integrity of the input data and then ensuring the accuracy of the output results.

[0160] The above description is based on the ideal embodiment of the present invention. Through the above description, relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A fusion method based on TextRAG and VisRAG, characterized in that: The following steps are involved: S1. Question segmentation and encoding. Question segmentation refers to: segmenting the question using a character-level segmentation method to obtain a character-level word array; S2, document loading and image conversion, according to the different types of documents in the knowledge base, the corresponding loading method is used to convert the documents into text form; S3, document information enhancement: Use text segmentation method to segment text documents into text blocks, then use LLMs to generate text block tables and hypothetical query questions, then concatenate the title, hypothetical query and text content, perform character-level segmentation again, and then remove stop words, so that each text block can get a phrase; S4, document encoding, concatenate the text title, hypothetical query, and text content of each text block, segment the text block into words using character-level segmentation, and encode each word after removing stop words; S5, similarity calculation, by calculating the similarity between the question encoding and all the text block encodings in the knowledge base, the top 10 text blocks most relevant to the question are screened out; S6, Top-k text image retrieval; S7, multimodal question answering; S8. Answer reference generation: adding reference tags to LLMs’ answers to increase the accuracy of knowledge question answering results.

2. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: The S1 comprises the following steps: S1-1, use jieba word segmenter to segment the question into individual words; S1-2, based on a commonly used stop word, remove the stop word to obtain a word array Q; S1-3. Use a deep learning text encoder to encode each word into a 768-dimensional numerical vector 3. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: In S1-2, the expression of word array Q is: Q=Q1Q2...Q n ; In S1-3, the i-th word Q in the question after removing stop words i Numeric vector of The expression is: The matrix V consisting of the encodings of all words Q The expression is: Where: Q i Represents the i-th word in the question after removing stop words.

4. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: In S2, the document loading function provided by the open source tool langchain is used to support documents in doc, docx, pdf, markdown, csv, xslx, txt, HTML, and PPT formats; Document imaging means converting each page of a document into an image and saving it to a document image database to facilitate subsequent multi-modal large-model question answering. Document imaging is the operation of converting each page of a document in various formats into an image.

5. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: The S3 comprises the following steps: S3-1, preset the maximum length of a text block and the maximum overlapping length between adjacent text blocks, and set the key characters for segmentation to "\n", ".", ",", ".", "\t"; S3-2. Call the RecursiveCharacterTextSplitter method of langchain to split each text block.

6. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: The S4 comprises the following steps: S4-1. Connect the text title, hypothetical query, and text content in the following order: "Title: {text title} Query: {hypothetical query} Body: {text content}"; S4-2, using the jieba word segmenter to segment the concatenated content, and then using a common stop word list to remove stop words to obtain a word array D; S4-3. Convert each word in the word array D into a 768-dimensional numerical vector using the m3e text encoding model All words form a numerical matrix V D ; In S4-2, the expression of word array D is: D=D1D2...D m ; Where: D j Represents the jth word in the document after removing stop words In S4-3, the numerical vector The expression is: Numerical matrix V D The expression is:

7. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: The S5 comprises the following steps: S5-1. For the problem encoding matrix V Q and the text block encoding matrix V D , calculate the MaxSim value of question Q and text block D as the text similarity between question Q and text block D; S5-2, for all text blocks D1, D2, ..., D in the knowledge base v , calculate the MaxSim value of each of them and question Q respectively to obtain scores s1, s2, ..., s v ; S5-3, for s i Sort by size and filter out the top 10 s i The corresponding text blocks D1, D2, ..., D 10 ; Where: s j D j and the MaxSim score of question Q; In S5-1, the calculation formula of MaxSim value is:

8. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: The S6 comprises the following steps: S6-1. According to the top-10 text blocks extracted in S5, locate the corresponding document and the document image corresponding to the page number of the document through its attributes doc_id and page_num. Assume that the corresponding document images are P1, P2, ..., P 10 ; S6-2, delete duplicate document images according to doc_id and page number page_num to obtain document images P1, P2, ..., P n ; Where: n≤10.

9. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: In S7, the document images P1, P2, ..., P corresponding to the question and the top 10 most relevant text blocks are displayed. n By constructing the input of VLMs, the output of VLMs is the answer to the given problem.

10. The fusion method based on TextRAG and VisRAG according to claim 1, characterized in that: The S8 comprises the following steps: S8-1, using ".", ", ", "!", "?", "\n" as the end symbol of Orange, segment the LLMs' answers at the sentence level; S8-2. Use the Jieba word segmenter to segment each sentence, remove stop words, and then perform word-level encoding to obtain the data matrix V of each sentence S. S ; S8-3, calculate the MaxSim value of sentence S and the 10 related text blocks obtained in S5. If these 10 related text blocks D1, D2, ..., D 10 If the maximum MaxSim value is greater than the preset threshold value 0.7, execute S8-4, otherwise, execute S8-5; S8-4, it is considered that the sentence needs to be marked with a reference tag, and the reference tag is the sequence number of the largest text block of MaxSim in these 10 related text blocks; S8-5, believes that the sentence only plays an excessive role and does not need a reference label; Where: The data matrix V of each sentence S S The expression is:

Images