Retrieval augmented generation knowledge base processing, retrieval augmented generation method, and electronic device

Abstract

This application discloses a method and electronic device for retrieval enhancement and knowledge base generation. The method includes: performing structured parsing and format reconstruction on an original knowledge document to convert it into a target document; rendering the target document into a target image; segmenting the target image into a sequence of image blocks, and visually encoding the image blocks using a visual encoding model to generate visual knowledge vectors based on visual encoding; constructing an index based on the visual knowledge vectors to form an external knowledge base for an artificial intelligence (AI) model, so that upon receiving query text, the query text is converted into an image using the same method as the target document and encoded into a visual query vector based on the visual encoding model; and the visual query vectors are used to retrieve matching visual knowledge vectors from the external knowledge base to enhance the AI ​​model's response content generation. This application embodiment can improve retrieval efficiency while ensuring semantic fidelity.

Description

Technical Field

[0001] This application relates to the field of AI generation technology, and in particular to retrieval enhancement generation knowledge base processing, retrieval enhancement generation methods, and electronic devices. Background Technology

[0002] With the widespread application of AI (Artificial Intelligence) models in enterprise-level knowledge management, intelligent customer service, document understanding, and other scenarios, RAG (Retrieval-Augmented Generation) technology has become a key path to improve the factual accuracy and domain adaptability of these models. This RAG technology introduces an external knowledge base into the AI ​​model, dynamically retrieving knowledge fragments relevant to the user's query during the inference phase and injecting them as context into the AI ​​model, thereby compensating for the limitations of the parameterized knowledge within the AI ​​model itself.

[0003] In a typical RAG system, the external knowledge base is usually constructed from unstructured or multimodal documents uploaded by users, including but not limited to PDF documents, PPT presentations, scanned images, Excel spreadsheets and their embedded charts. To support efficient retrieval, these raw documents need to be preprocessed and converted into vector form before being stored in the index. However, current mainstream methods generally adopt a workflow of "text extraction → fixed-length chunking → text vectorization," which faces at least the following core challenges when processing complex enterprise-level documents: The loss of semantic structure leads to decreased retrieval accuracy: Due to the lack of a unified semantic modeling mechanism for documents of different formats, the parsing process often ignores the hierarchical structure of the original document (such as chapter titles, nested lists, code blocks, and reference relationships). Especially in scenarios with mixed text and images, images are often treated as attachments, resulting in a break in the semantic connection with adjacent text. Furthermore, mechanically segmenting text into blocks based on the number of characters easily leads to the fragmentation of semantic units (for example, placing a question and its answer in different blocks), severely weakening the contextual coherence and relevance of the retrieval results.

[0004] Second, the proliferation of context tokens leads to a surge in inference costs: To retain sufficient information, original documents are often segmented into a large number of highly redundant text blocks, resulting in a sharp increase in the number of context tokens input to the AI ​​model. On the one hand, long contexts are prone to exceeding the model window limit, causing key information to be truncated; on the other hand, repetitive text with low information density significantly increases computational overhead and inference latency, making RAG systems face high deployment and operation costs in large-scale knowledge base scenarios.

[0005] Although the industry has tried to optimize the RAG process by improving document parsers, introducing structured tags, or compressing vectors, existing solutions are still limited to a "text-centric" processing paradigm: they either focus only on information extraction while ignoring end-to-end compression, or they compress vector storage but cannot reduce the actual number of tokens fed into the AI ​​model.

[0006] Therefore, there is an urgent need for a new knowledge base construction method that can simultaneously ensure semantic integrity, achieve cross-modal unified representation, and significantly reduce context token consumption, in order to support high-precision, low-cost, and scalable enterprise-level RAG applications. Summary of the Invention

[0007] This application provides methods and electronic devices for AI model knowledge base processing and retrieval enhancement generation, which can improve retrieval efficiency and reduce retrieval costs while ensuring semantic fidelity.

[0008] This application provides the following solution: A retrieval enhancement knowledge base generation method includes: The original knowledge document is subjected to structured parsing and format reconstruction to be converted into a target document, so that the target document retains the complete semantics and structured information of the original knowledge document, and the layout information density of the target document is higher than that of the original knowledge document. Render the target document into a target image; The target image is segmented into a sequence of image blocks, and the image blocks are visually encoded using a visual encoding model to generate a visual knowledge vector based on visual encoding. Based on the visual knowledge vectors, an index is constructed to form an external knowledge base for the artificial intelligence (AI) model. After receiving the query text, the query text is converted into an image using the same method as the target document. Then, it is encoded into a visual query vector based on a visual encoding model. Finally, the visual query vector is used to retrieve matching visual knowledge vectors from the external knowledge base to enhance the AI ​​model's response content generation.

[0009] The structured information from the original knowledge document retained in the target document includes: The original knowledge document contains chapter levels, paragraphs, lists, code, and / or reference information. The original knowledge document includes documents that combine images and text; The structured information retained in the original knowledge document in the target document includes: The target document retains the text, images, hyperlinks, corresponding spatial layouts, and / or semantic relationships from the original knowledge document.

[0010] The target document is a text-formatted document; When reconstructing the original knowledge document into a structured format to convert it into a target document, the images in the original knowledge document are converted into link addresses, and the anchor position of the link address in the target document is determined. When rendering the target document into a target image, the corresponding image data is obtained according to the link address to restore the image, and the link address at the corresponding anchor point is replaced with the restored image content.

[0011] If the images in the original knowledge document contain text content, the method further includes: Before structured parsing and format reconstruction, optical character recognition (OCR) technology is used to identify the text content from the image, and the text recognition results and the original image are both retained in the target document.

[0012] This also includes: After converting the original knowledge document into a target document, the text content in the target document is semantically enhanced using an AI model, so that the target image can be rendered and visual vectors can be generated based on the semantically enhanced text content.

[0013] This also includes: After generating knowledge vectors based on visual encoding, the knowledge vectors are quantified to generate visual tokens, and a bidirectional mapping relationship is established between the visual tokens and the corresponding image blocks at anchor positions in the target document. The visual token is the smallest input unit of the AI ​​model. During the content generation process using the input knowledge tokens, the AI ​​model decodes multiple output tokens into image blocks, and according to the anchor positions corresponding to the multiple output tokens in the bidirectional mapping relationship, the decoded image blocks are spliced ​​together based on the anchor position information. The text content is then identified and output from the spliced ​​image.

[0014] A retrieval enhancement generation method, comprising: Receive query text information input by the user; The query text information is converted into an image using the same rendering method as in the knowledge base construction phase; After visually encoding the image using a visual encoder, a visual query vector based on the visual encoding is obtained. The visual query vector is used to retrieve matching visual knowledge vectors from an external knowledge base, and the retrieved target visual knowledge vectors are used as context input to the AI ​​model to enhance the AI ​​model's response content generation; the external knowledge base is established by the method described in any of the preceding methods.

[0015] The external knowledge base also stores visual tokens generated after quantifying visual knowledge vectors, as well as a two-way mapping relationship between visual tokens and anchor positions in the target document; the visual token is the smallest input unit of the AI ​​model. After retrieving the target visual knowledge vector, the process also includes: The visual query vector and the visual token corresponding to the target visual knowledge vector are input into the AI ​​model so that the AI ​​model outputs multiple visual tokens. The multiple output tokens of the AI ​​model are decoded into image blocks, and the anchor point position information of the multiple output tokens in the target document is obtained according to the bidirectional mapping relationship. After stitching the decoded image blocks according to the anchor point location information, the text content in the stitched image is identified and output as the generated answer.

[0016] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of any of the preceding methods.

[0017] An electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of any of the preceding methods.

[0018] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of any of the preceding methods.

[0019] According to the specific embodiments provided in this application, the following technical effects are disclosed: Through the embodiments of this application, the original knowledge document can first undergo structured parsing and format reconstruction to convert it into a target document. The target document retains the complete semantics and structured information of the original knowledge document, and its layout information density is higher than that of the original knowledge document. Then, the target document can be rendered into a target image, and the target image can be segmented into a sequence of image blocks. These image blocks are then visually encoded using a visual encoding model to generate visual knowledge vectors based on visual encoding. Next, an index is constructed based on these visual knowledge vectors to form an external knowledge base for the artificial intelligence (AI) model. Upon receiving query text, the query text is converted into an image using the same method as the target document, and encoded into a visual query vector using the visual encoding model. The visual query vector is then used to retrieve matching visual knowledge vectors from the external knowledge base, thereby enhancing the AI ​​model's response content generation. This approach encodes and generates vectors from the original knowledge document (which may include information from multiple modalities) under a unified visual modality. Although it involves image patch division, the visual modality encoding process involves convolutional processing of high-dimensional pixel data. This convolutional processing preserves relatively complete structured information between different image patches, thus solving the problem of modality and structured information loss and restoring the original information. Furthermore, "a picture is worth a thousand words," and referencing human memory methods, images and scenes are used for memorization, allowing problems that one-dimensional text cannot solve to be addressed in two-dimensional images, thereby improving retrieval accuracy. Additionally, because compression is performed during the conversion of the original knowledge document into the target document, the target document retains the complete semantic and structured information of the original knowledge document while improving information utilization. Therefore, the content on the same page, which might have thousands of tokens using text-based encoding vectors, only has hundreds of visual tokens after converting the target document, rendering it as an image, and generating visually encoded vectors. Therefore, the number of tokens in the external knowledge base can be significantly reduced. When performing vector retrieval based on user queries, the retrieval can be performed within a smaller vector range. Consequently, the number of vector similarity calculations is also greatly reduced, thereby improving retrieval efficiency and reducing retrieval costs.

[0020] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the system architecture provided in the embodiments of this application; Figure 2 This is a flowchart of the first method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the first interface provided in an embodiment of this application; Figure 4 This is a schematic diagram of the second interface provided in an embodiment of this application; Figure 5 This is a flowchart of the second method provided in the embodiments of this application; Figure 6 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0024] First, to facilitate understanding of the technical solutions provided in the embodiments of this application, we will first give a brief introduction to the concepts of Token and related concepts in the AI ​​model.

[0025] Tokens are the basic units for models to process data. Taking an AI language model (where both input and output are text content) as an example, the input text needs to be segmented into a token sequence by a token segmenter, and then mapped into a vector representation for the model to compute. The number of tokens directly determines the context length, computational overhead, and inference cost. For example, a document containing 3000 Chinese characters typically corresponds to about 3000–6000 tokens, which can easily exceed the capacity of mainstream models (such as a 32K context window).

[0026] As can be seen from the above introduction to the meaning and function of Tokens, ordinary knowledge documents cannot be directly input into AI models and require prior processing. Therefore, when creating an external knowledge base for an AI model, the existing technology typically handles the following: First, the original knowledge document is input (e.g., a user-uploaded document). Since the information content of a single file can be quite large, and the length of the input vector that an AI model can accept is limited, the current processing method typically involves dividing text content into multiple text blocks of fixed character length, and image content into multiple image blocks based on regions. Each block is then vectorized, and the resulting vectors are indexed and saved to an external knowledge base. Specifically, during retrieval and generation, the similarity between the query vector and vectors in the knowledge base is calculated to retrieve the top 5 (e.g., 5) most similar vectors. The AI ​​model then uses the query vector and these retrieved vectors together to generate the response. However, this approach has a problem: dividing the text content into blocks of fixed character length may cause semantic truncation. Furthermore, if a single file contains multiple modalities, such as both text and images, content from other modalities may be lost. Alternatively, other modalities can be pre-extracted, but this requires separate vectorization of each modality. However, this often results in a focus on text, with images and lists typically treated as attachments. This leads to significant loss of structured information in the original file, causing a disconnect between text and images. These issues make it difficult to fully and accurately represent the knowledge in the original file using vectors from the knowledge base. Consequently, the accuracy of retrieval during the augmented reality generation process is insufficient, impacting the quality of the final generated results.

[0027] Another optimization approach is vector compression. This involves compressing the high-dimensional vectors obtained after vectorizing the segmented results into low-dimensional vectors – essentially dimensionality reduction. However, this method only reduces the dimension of each vector; it doesn't reduce the number of vectors or corresponding tokens in the knowledge base. Therefore, the number of matching operations remains the same, limiting the cost savings from vector compression. Furthermore, dimensionality reduction can lead to semantic distortion, potentially affecting search accuracy.

[0028] To address the above issues, this application provides a corresponding solution. This solution first involves standardizing and parsing the input original knowledge document, converting it into a target document in a specific format (e.g., Markdown). During the conversion process, while preserving the complete semantics and structured information of the original knowledge document, certain processing methods are used to make the target document's page utilization rate higher than the original document. That is, the target document's content is more compact and its information density is higher. For example, content that would require 10 or more pages in the original knowledge document may only need two or three pages in the converted target document. The specific structured information may include chapter levels, paragraphs, lists, code, and / or citation information. In cases where the original knowledge document contains mixed text and images, information such as the images and their positional relationships with the text can also be preserved.

[0029] After converting the original knowledge document into a target document, this target document can be rendered as an image, such as a picture in a specific format. For example, the original document might be in Word, PDF, or other formats, while the target document might be in Markdown or other formats. Then, the Markdown document can be rendered into an image format.

[0030] After rendering the target document into an image, the target image can be segmented into a sequence of image blocks. These image blocks are then visually encoded using a visual encoder to generate visual vectors. These visual vectors can then be used to build an index and save it as an external knowledge base for the AI ​​model.

[0031] In other words, in this embodiment, the external knowledge base stores visual vectors. Even if the original knowledge document includes text content, it is rendered into an image and then a visual vector (which can be called a vision-based knowledge vector) is generated and stored. Optionally, the visual token corresponding to the visual vector can also be stored in the external knowledge base. When a user's query information is received, this query information is usually expressed in text form. In this case, the user's query text can first be rendered into an image. Then, the image can be encoded using the same visual encoding model used when creating the knowledge base to generate a vision-based query vector. Then, through "image search," several knowledge vectors that meet the similarity criteria to the query vector can be retrieved from the knowledge base (including the AI ​​model's internal knowledge base and the external knowledge base). Furthermore, the knowledge vectors retrieved from the external knowledge base can enhance the AI ​​model's knowledge, thereby enabling the AI ​​model to generate the response content.

[0032] It should be noted that the AI ​​model used in this embodiment can be an AI language model. The basic function of such an AI language model is typically text generation based on text. That is, theoretically, the input to this AI language model is a text token, and the corresponding output is also a text token. However, in this embodiment, the vectors stored in the external knowledge base are generated visually. This means that the input to the AI ​​language model needs to be changed from text tokens to visual tokens. Since both are token-level inputs, the AI ​​language model can support this type of input. Of course, visual tokens and text tokens may have some differences in nature. For example, because they come from different feature spaces, the numerical distribution of the tokens may be different; in addition, since text tokens usually have explicit semantics, while visual tokens are mainly abstract features, they also differ in semantic granularity. Therefore, in specific implementations, projection alignment can be performed before inputting the visual tokens into the AI ​​language model; that is, the visual tokens can be "translated" into language that the AI ​​language model can understand. Specifically, this may include... Dimension matching: Mapping the number of dimensions of visual features to the number of dimensions of text vectors in an AI language model; Distribution alignment: adapting visual features to the input distribution expected by the AI ​​language model; Semantic alignment: learning to map visual concepts to linguistic concepts.

[0033] Alternatively, the specific AI model could not be an AI language model, but rather use a more efficient fusion architecture. In this case, there is no need to project and align visual tokens; they can be encoded directly in a unified space. Furthermore, text, images, and audio can be expressed using the same token, achieving a more unified multimodal model, and so on.

[0034] In the above methods, see Figure 1First, the original knowledge document is transformed into a more efficient and compact target document (specifically, a document in Markdown or similar format), while retaining the complete semantics and structured information of the original knowledge document. Next, the target document is rendered as an image, segmented, and then visually encoded knowledge vectors are generated. A vector index is then constructed and saved to an external knowledge base. Subsequently, during specific queries, the query text can be converted into an image and its corresponding visual query vector, enabling vector retrieval through "image search," which then aids in AI model enhancement. This approach encodes and generates vectors from the original knowledge document (which may include information from multiple modalities) under a unified visual modality. Although it involves image patch division, the visual modality encoding process involves convolutional processing of high-dimensional pixel data. This convolutional processing preserves relatively complete structured information between different image patches, thus solving the problem of modality and structured information loss and restoring the original information to the greatest extent. Furthermore, "a picture is worth a thousand words." Referring to human memory methods, images and scenes are used for memorization, allowing two-dimensional images to solve problems that one-dimensional text cannot, thereby improving retrieval accuracy. Additionally, the same page of content might have thousands of tokens using text-based vector encoding, but after being converted into images and generated as visually encoded vectors, the number of visual tokens is only in the hundreds. This significantly reduces the number of tokens in the external knowledge base. When performing vector retrieval based on user queries, the search can be conducted within a smaller vector range, correspondingly reducing the number of vector similarity calculations, thereby improving retrieval efficiency and reducing retrieval costs.

[0035] In summary, the original multimodal knowledge document is structurally parsed and reconstructed into a semantically complete target document. Then, the target document is rendered into a high-fidelity target image and segmented into a sequence of image blocks. Each image block is then encoded using a visual encoding model to generate a compact visual knowledge vector. An index is built based on these visual knowledge vectors to form an external knowledge base for the AI ​​model. During the query phase, the user-input text query is converted into a query image using the same rendering strategy and then processed by the same visual encoding model to generate a visual query vector. Relevant knowledge fragments are retrieved from the knowledge base using near-nearest neighbor retrieval to enhance the generation process of the large language model. This approach breaks through the traditional text-to-token (TTP) processing paradigm. Through an end-to-end process of "structural reconstruction → image rendering → visual token compression," it achieves a high information compression rate while preserving the original document's hierarchical structure, text-image layout, and semantic relationships, significantly reducing the AI ​​model's context token consumption and improving retrieval accuracy and question-answering generation quality. Furthermore, this solution innovatively transforms the "text-to-text" task into a "visual-to-text" task. By introducing the concept of "pixel as context" information compression and drawing on the human ability to better remember images and scenes, this method utilizes two-dimensional spatial layout to retain layout, hierarchy, and relational information that is difficult for one-dimensional linear text to contain. This significantly reduces token consumption while improving semantic integrity and contextual coherence, achieving truly efficient multimodal knowledge representation and RAG enhancement.

[0036] The specific implementation schemes of the embodiments of this application will be described in detail below.

[0037] Example 1 First, this first embodiment addresses the specific process of building an index and creating an external knowledge base by providing a retrieval-enhanced knowledge base generation method, see [link to relevant documentation]. Figure 2 The method may specifically include: S201: Perform structured parsing and format reconstruction on the original knowledge document to convert it into a target document, such that the target document retains the complete semantics and structured information of the original knowledge document, and the layout information density of the target document is higher than that of the original knowledge document.

[0038] The original knowledge document can be input by the user, including ordinary individual users or users in enterprise scenarios. For individual users, they can upload a local file as the original knowledge document while asking questions to the AI ​​model, allowing the AI ​​model to generate content based on this document. For example, the main interface of the AI ​​model application can provide interactive entry points such as dialog boxes, through which users can input specific query text. Additionally, the lower right corner of the dialog box provides options for uploading documents, allowing users to select the document to upload from their local folder. In enterprise application scenarios, knowledge base management functions can be provided in the AI ​​model's management interface. Enterprise users can initiate the creation of external knowledge bases through this interface, uploading internal company documents as the basic knowledge document, and so on.

[0039] After receiving the original knowledge document, structured parsing and format reconstruction can be performed to convert it into the target document. In practice, the format and parsing path can be determined before structured parsing and format reconstruction. For documents with mixed modal content, layout structure, body text, titles, figure captions, table captions, etc., can be extracted. Additionally, scanned image text can be processed, establishing relationships between image and text regions, and outputting structured text, layout metadata, and initial image / table recognition results, etc. After completing the above preprocessing, structured format reconstruction is performed. For example, if the target document is in Markdown format, the structured reconstruction process can annotate six levels of headings (H1-H6), lists, code, citations, etc., to reconstruct the original knowledge document into a semantically complete Markdown document.

[0040] It's important to note that if the original knowledge document is a mix of images and text, the structured information retained in the target document from the original knowledge document will include: the text, images, hyperlinks, corresponding spatial layout, and / or semantic relationships from the original knowledge document. In other words, the images and text from the original knowledge document will be preserved in the target document, and the relationship between images and text will also be retained. For example, if a piece of text explains an image, that information will also be retained in the target document. Of course, the positional relationship between images and text may not be entirely consistent with that in the original knowledge document. For instance, suppose in the original knowledge document there is text below an image explaining that image, and there is a large blank area to the right of the image. In the target document, this text might appear to the right of the image for space utilization considerations, and so on.

[0041] Of course, since the target document is usually a plain text document, when reconstructing the original knowledge document into a structured format to convert it into the target document, the images in the original knowledge document can first be converted into link addresses (the image files extracted from the original knowledge document can be saved to a cache directory during the preprocessing stage, and the link address can be the storage address of the image file in the cache directory), and the anchor position of the link address in the target document can be determined. That is, in the specific target document, the anchor position is the link address of the image. Subsequently, when rendering the target document into the target image, the corresponding image data can be obtained first according to the link address to restore the image, and the link address at the corresponding anchor position can be replaced with the restored image content.

[0042] If the images in the original knowledge document contain text, OCR (Optical Character Recognition) technology can be used during structured reconstruction to identify the text from the images, and both the text recognition results and the original image are retained in the target document. If the images in the original knowledge document contain tables, the text content and table structure information can also be identified from the tables during structured reconstruction. This table structure information can then be retained in the target document; for example, the text in the table can be expressed as multiple rows and columns, but table cells and other elements will not be directly present.

[0043] Furthermore, during the reconstruction process, certain processing can be performed to increase the information density of the target document compared to the original knowledge document. This means that the target document requires fewer pages to express the same semantic content than the original knowledge document. For example, specific processing methods may include reducing the image size and font size in the original knowledge document, shrinking the images or text to a size that the model can accurately recognize, thereby reducing the space occupied. Additionally, redundant elements and / or formatting in the original knowledge document can be removed, or some blank areas in the original knowledge document can be fully utilized, and so on. Through these processing methods, the complete semantic and structured information of the original knowledge document can be preserved in the target document, while also allowing more information to be expressed within the same page area; that is, the content is more compact and the information density is higher. For example, content that requires 10 or more pages in the original knowledge document may only need two or three pages in the converted target document. The specific structured information may include chapter levels, paragraphs, lists, code and / or reference information. In the case of multimodal content mixed in the original knowledge document, it may also include information such as images, text and the relationship between images and text.

[0044] Furthermore, optionally, after converting the original knowledge document into a target document, an AI model can be used to semantically enhance the text content of the target document. This enhanced text content can then be used to render the target image and generate visual vectors. Specific semantic enhancement can include further supplementing the text content in the target document, adding relevant explanations, or, for text content identified from tables, adding relevant separator elements, and so on. Of course, these separator elements are expressed using forms supported by the target document; for example, the specific separator element may not be a "line" element but a symbolic element, and so on.

[0045] For example, such as Figure 3 As shown, point 31 shows a portion of the original knowledge document, point 32 shows the state after being converted to Markdown, and point 33 shows the state after being semantically enhanced by an AI model.

[0046] For example, such as Figure 4 As shown, suppose the original knowledge document is an image that includes text and tables, such as... Figure 4 As shown at point 41; in this case, the text can first be identified using methods such as OCR recognition, including the text in the table. For example, the recognition result can be as follows: Figure 4 As shown at point 42 in the image. Then, the OCR results can be enhanced using an AI model; the specific output can be shown as follows... Figure 4 As shown at point 43 in the table, it can be seen that the text identified in the table can be supplemented with separators to better preserve the original table structure. Additionally, it's possible to add interpretations of the specific text content within the table. It's worth noting that, under the preferred method, the target document can be converted to Markdown format. This allows for better structuring and semantic fidelity of knowledge across formats, and improves page layout utilization. Specifically, this can be seen in the following aspects: 1. Unified Layout and Structure: Markdown provides a strict hierarchical structure (headings, lists, tables, code, images, etc.), eliminating noise in fonts, pagination, and layout of the original format, and improving the parsing accuracy of the multimodal model and the comparability between documents.

[0047] 2. Version and Reference Management: Using Markdown in conjunction with professional document management tools, documents can be uniformly managed and version controlled, ensuring clear links between documents, reducing redundancy and information conflicts, and making cross-document retrieval clearer.

[0048] 3. Facilitates information compression and image processing: The structural integrity of image content is a prerequisite for information compression. Markdown has no pagination limitations and supports a unified rendering template, improving the accuracy of image and text content extraction and laying the foundation for subsequent high-fidelity compression.

[0049] 4. Adapts to natural language and multimodal conversion: Markdown's hierarchical structure is naturally compatible with multimodal models, supporting bidirectional high-precision conversion between images and text, improving the quality of retrieval and question answering.

[0050] S202: Render the target document into a target image.

[0051] After obtaining the target document, it can be rendered into a target image, such as a picture. There are various rendering strategies available, depending on the page length and layout complexity of the target document. For example, for shorter pages with simpler layouts, a standard rendering strategy can be used to achieve high DPI (Dots Per Inch) rendering per page. Furthermore, for such short documents, multiple short documents can be merged for rendering. For instance, suppose there are multiple original knowledge documents, each converted into multiple target documents, but each target document is relatively short. In this case, multiple short documents can be rendered into the same image to achieve maximum compression, and the content from multiple documents can be merged into a single image for maximum integration. For example, in... Figure 1 The example shown illustrates this short document scenario. As can be seen from the image, after converting the three original knowledge documents into Markdown documents separately, the three Markdown documents are merged and rendered into a single image, and then visual encoding is performed. It should be noted that... Figure 1 It is mainly used to express the specific processing flow and the output of each stage. You do not need to pay attention to the specific text content in the document.

[0052] For long pages with complex layouts containing formulas, tables, and other dense elements, using standard rendering strategies may result in information loss or blurring. Therefore, enhanced rendering strategies can be adopted, such as enabling multi-scale strategies and tiling strategies.

[0053] The so-called multi-scale strategy refers to rendering at different granularities based on the actual situation, rather than using the highest resolution for all details. For example, the overall layout can be rendered at a lower resolution using scale 1, chapter-level elements can be magnified using scale 2, and text and formula details can be magnified at a higher resolution using scale 3, and so on. In this way, both the overall layout and local details can be taken into account, making smaller text and formulas in complex layouts clearer and more legible in the rendered image.

[0054] The so-called slicing strategy refers to the practice of slicing long documents before rendering them. This is because rendering engines may have maximum resolution limitations, and there are also memory limitations (extremely large images may not be able to be loaded into the GPU). Therefore, the original long document can be sliced ​​before rendering. It is clear that by slicing and segmenting extremely long documents, the aforementioned technical limitations can be overcome.

[0055] S203: The target image is segmented into a sequence of image blocks, and the image blocks are visually encoded using a visual encoding model to generate a visual knowledge vector based on visual encoding.

[0056] After rendering the target document into a target image, visual encoding can be performed. However, before encoding, the target image can first be segmented into a sequence of image blocks. Then, visual encoding is performed on each image block. Visual encoding of the image blocks yields vectors. Since these vectors are based on visual encoding, and the target image is rendered after being transformed from the original knowledge document, primarily expressing knowledge information, the vector obtained in this step is called a "visual knowledge vector" to distinguish it from other vectors obtained later.

[0057] It should be noted that although the target image is divided into multiple image blocks and then encoded separately, the visual encoding process involves convolution processing of high-dimensional pixel data. This convolution processing can preserve relatively complete structural information between different image blocks. Therefore, it can solve the problem of loss of modality and structural information and restore the original information to the greatest extent.

[0058] S204: Based on the visual knowledge vector, an index is constructed to form an external knowledge base for the artificial intelligence (AI) model. After receiving the query text, the query text is converted into an image using the same method as the target document. After being encoded into a visual query vector based on the visual encoding model, the visual query vector is used to retrieve matching visual knowledge vectors from the external knowledge base to enhance the generation of the AI ​​model's response content.

[0059] After obtaining the visual knowledge vectors based on visual encoding, an index can be built based on these vectors to form an external knowledge base for the AI ​​model. Thus, when generating response content based on the user's query text, the query text can first be converted into an image using the same method as the target document, and then encoded into a visual query vector based on the visual encoding model. The visual query vector can then be used to retrieve matching visual knowledge vectors from the external knowledge base, thereby enhancing the knowledge of the AI ​​model and generating the response content. In other words, in this embodiment, since both the vectors in the external knowledge base and the query vectors are visually encoded, the vector retrieval is a "search by image" method.

[0060] It should be noted that in practical implementation, after the matching knowledge vector is searched and input into the AI ​​model, it is usually the token corresponding to the vector (the smallest input unit of the AI ​​model). As mentioned earlier, in some implementations, the vector obtained by visual encoding can be directly used as the token, while other implementations require further quantization of the vector obtained by visual encoding to obtain the visual token. If further quantization of the vector obtained by visual encoding is required, this quantization process can also be completed when creating the external knowledge base. Therefore, in the optional implementation of this application, after generating the knowledge vector based on visual encoding, the knowledge vector can be quantized to generate a visual token, and this visual token can also be saved in the external knowledge base.

[0061] Furthermore, since the AI ​​model's generated results are also visual tokens, it calculates the probability distribution of each token in its vast token set as the "next token" based on the input visual tokens and the already generated tokens. Then, it selects a token for output based on this probability distribution, and so on. However, because the final output needs to be text-based, these output visual tokens must be converted back into image blocks, stitched together, and then the text content must be identified from the image before the final text-based response can be output. During the stitching of image blocks, the positional information of the image blocks is also needed, which can be obtained from the anchor point positions in the target document. Therefore, after generating visual tokens, a two-way mapping relationship can be established between visual knowledge tokens and anchor point positions in the target document. That is, during the conversion of the original knowledge document into the target document, anchor point position information for specific text content is generated in the target document, and this anchor point position information can be used to locate the specific text content within the target document. After converting the target document into a target image and then segmenting it into image blocks, a correspondence exists between specific image blocks and the text content in the target document. Correspondingly, the anchor position of the text content corresponding to the image block in the target document can also serve as the anchor position of the image block within the target document. Thus, after the AI ​​model generates content using the input knowledge tokens, it can decode multiple output tokens into image blocks. Based on the bidirectional mapping relationship and the anchor positions of the image blocks corresponding to the multiple output tokens, the decoded image blocks are stitched together. The text content is then identified and output from the stitched image.

[0062] In summary, through the embodiments of this application, the original knowledge document can first be structured and reconstructed to convert it into a target document, ensuring that the target document retains the complete semantics and structured information of the original knowledge document, and that the layout information density of the target document is higher than that of the original knowledge document. Then, the target document can be rendered into a target image, and the target image can be segmented into a sequence of image blocks. After visually encoding the image blocks using a visual encoding model, visual knowledge vectors based on visual encoding are generated. Then, an index is constructed based on the visual knowledge vectors to form an external knowledge base for the artificial intelligence (AI) model. Upon receiving query text, the query text is converted into an image using the same method as the target document, and encoded into a visual query vector based on the visual encoding model. The visual query vector is then used to retrieve matching visual knowledge vectors from the external knowledge base, thereby enhancing the AI ​​model's response content generation. This approach encodes and generates vectors from the original knowledge document (which may include information from multiple modalities) under a unified visual modality. Although it involves image patch division, the visual modality encoding process involves convolutional processing of high-dimensional pixel data. This convolutional processing preserves relatively complete structured information between different image patches, thus solving the problem of modality and structured information loss and restoring the original information. Furthermore, "a picture is worth a thousand words," and referencing human memory methods, images and scenes are used for memorization, allowing problems that one-dimensional text cannot solve to be addressed in two-dimensional images, thereby improving retrieval accuracy. Additionally, because compression is performed during the conversion of the original knowledge document into the target document, the target document retains the complete semantic and structured information of the original knowledge document while improving information utilization. Therefore, the content on the same page, which might have thousands of tokens using text-based encoding vectors, only has hundreds of visual tokens after converting the target document, rendering it as an image, and generating visually encoded vectors. Therefore, the number of tokens in the external knowledge base can be significantly reduced. When performing vector retrieval based on user queries, the retrieval can be performed within a smaller vector range. Consequently, the number of vector similarity calculations is also greatly reduced, thereby improving retrieval efficiency and reducing retrieval costs.

[0063] Example 2 The first embodiment described above mainly introduced the process of creating an external knowledge base. This second embodiment mainly introduces the method of enhancing the retrieval of the external knowledge base created in the above manner. Specifically, this second embodiment provides a retrieval enhancement generation method, see [link to relevant documentation]. Figure 5 The method may specifically include: S501: Receive query text information input by the user.

[0064] The specific query text can be entered by the user or received from other upstream systems. The query text is essentially the textual expression of the query requirement, which can specify the content to be generated.

[0065] S502: Convert the query text information into an image using the same rendering method as in the knowledge base construction phase.

[0066] In this embodiment of the application, since the external knowledge base stores vision-based knowledge vectors, after receiving the query text, the query text can also be rendered as an image first, so as to perform "image search".

[0067] S503: After visually encoding the image using a visual encoder, a visual query vector based on the visual encoding is obtained.

[0068] Before performing a search, the image can be visually encoded to obtain a query vector based on the visual encoding. Query text is usually not very long, so it can be directly visually encoded after being converted into an image. If the query text is long, and the converted image size exceeds the maximum size of a single image patch, the image can be divided into patches first, and then visually encoded to obtain multiple query vectors.

[0069] S504: Using the visual query vector, retrieve matching visual knowledge vectors from an external knowledge base, and input the retrieved target visual knowledge vectors as context into the AI ​​model to enhance the AI ​​model's response content generation; the external knowledge base is established using the method described in Embodiment 1.

[0070] After obtaining the visual query vector, the visual knowledge vector can be used to retrieve visual knowledge vectors from an external knowledge base. For example, multiple target visual knowledge vectors with the highest similarity can be retrieved. These target visual knowledge vectors can then be used to enhance the internal knowledge of the AI ​​model, thereby helping the AI ​​model generate response content.

[0071] In a specific implementation, the external knowledge base may further include visual tokens generated after quantifying the knowledge vectors, and a bidirectional mapping relationship between the visual tokens and anchor positions in the target document. Thus, after retrieving the target visual knowledge vector, the visual query vector and the corresponding visual tokens can be input into the AI ​​model, causing the AI ​​model to output multiple visual tokens. Subsequently, the multiple output tokens of the AI ​​model can be decoded into image patches, and based on the aforementioned bidirectional mapping relationship, the anchor position information of the multiple output tokens in the target document can be obtained. After stitching the decoded image patches according to the anchor position information, the text content within the stitched image is identified and output as the generated answer.

[0072] For the parts not described in detail in the above embodiment two, please refer to the description in embodiment one and other parts of this specification, which will not be repeated here.

[0073] It should be noted that the embodiments of this application may involve the use of user data. In practical applications, user-specific personal data may be used in the scheme described herein within the scope permitted by applicable laws and regulations, provided that it complies with the applicable laws and regulations of the country (e.g., with the user's explicit consent, with the user being properly notified, etc.).

[0074] Corresponding to Embodiment 1, this application also provides a retrieval enhancement knowledge base generation processing apparatus, including: The structured reconstruction unit is used to perform structured parsing and format reconstruction on the original knowledge document to convert it into a target document, so that the target document retains the complete semantics and structured information of the original knowledge document, and the layout information density of the target document is higher than that of the original knowledge document. A rendering unit is used to render the target document into a target image; A visual encoding unit is used to segment the target image into a sequence of image blocks, and then use a visual encoding model to visually encode the image blocks to generate a visual knowledge vector based on visual encoding. The index building unit is used to build an index based on the visual knowledge vector and form an external knowledge base for the artificial intelligence (AI) model. After receiving the query text, the query text is converted into an image using the same method as the target document, and encoded into a visual query vector based on the visual encoding model. Then, the visual query vector is used to retrieve matching visual knowledge vectors from the external knowledge base to enhance the generation of response content by the AI ​​model.

[0075] The structured information from the original knowledge document retained in the target document includes: The original knowledge document contains chapter levels, paragraphs, lists, code, and / or reference information. If the original knowledge document includes a document with mixed images and text, the structured information retained in the target document from the original knowledge document may include: the text, images, hyperlinks, corresponding spatial layouts, and / or semantic relationships retained in the target document from the original knowledge document.

[0076] The target document is a text-formatted document; the device may further include: The address conversion unit is used to convert images in the original knowledge document into link addresses and determine the anchor position of the link address in the target document when reconstructing the original knowledge document into a structured format to convert it into a target document. The image restoration unit is used to obtain the corresponding image data according to the link address to restore the image when rendering the target document into a target image, and to replace the link address at the corresponding anchor point with the restored image content.

[0077] If the images in the original knowledge document contain text content, the device may further include: The OCR recognition unit is used to identify text content from the image using optical character recognition (OCR) technology before structured analysis and format reconstruction, and retains both the text recognition result and the original image in the target document.

[0078] Additionally, the device may also include: The semantic enhancement processing unit is used to perform semantic enhancement processing on the text content in the target document through an AI model after converting the original knowledge document into a target document, so as to render the target image and generate visual vectors based on the semantically enhanced text content.

[0079] Additionally, the device may also include: The token generation unit is used to generate visual tokens by quantifying the knowledge vectors after generating knowledge vectors based on visual encoding, and to establish a bidirectional mapping relationship between the visual tokens and the corresponding image blocks at anchor positions in the target document. The visual token is the smallest input unit of the AI ​​model. During the content generation process using the input knowledge tokens, the AI ​​model decodes multiple output tokens into image blocks, and according to the anchor positions corresponding to the multiple output tokens in the bidirectional mapping relationship, it splices the decoded image blocks based on the anchor position information, and then identifies and outputs the text content from the spliced ​​image.

[0080] Corresponding to Embodiment 2, this application also provides a retrieval enhancement generation device, which may include: The query text receiving unit is used to receive query text information input by the user. A rendering unit is used to convert the query text information into an image using the same rendering method as the knowledge base construction phase. The visual query vector generation unit is used to obtain a visual query vector based on the visual encoding after visually encoding the image using a visual encoder. The retrieval unit is used to retrieve matching visual knowledge vectors from an external knowledge base using the visual query vector, and inputs the retrieved target visual knowledge vectors as context into the AI ​​model to enhance the AI ​​model's response content generation; the external knowledge base is established using the method described in the aforementioned Embodiment 1.

[0081] The external knowledge base also stores visual tokens generated after quantifying visual knowledge vectors, and a bidirectional mapping relationship between the visual tokens and anchor positions in the target document; the visual token is the smallest input unit of the AI ​​model; at this time, the device may further include: The Token input unit is used to input the visual query vector and the visual token corresponding to the target visual knowledge vector into the AI ​​model after retrieving the target visual knowledge vector, so that the AI ​​model outputs multiple visual tokens. The decoding unit is used to decode multiple output tokens of the AI ​​model into image blocks, and obtain the anchor position information of the multiple output tokens in the target document according to the bidirectional mapping relationship. The post-processing unit is used to stitch the decoded image blocks according to the anchor point position information, and then identify the text content in the stitched image as the generated answer for output.

[0082] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0083] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0084] A computer program product includes a computer program / computer executable instructions that, when executed by a processor in an electronic device, implement the steps of the method described in the foregoing method embodiments.

[0085] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0086] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0087] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and a search enhancement generation processing system 625, etc. The aforementioned search enhancement generation processing system 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.

[0088] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0089] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0090] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0091] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0092] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0093] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0094] The foregoing has provided a detailed description of the retrieval enhancement knowledge base processing, retrieval enhancement generation method, and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application; furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for enhancing retrieval and generating a knowledge base, characterized in that, include: The original knowledge document is subjected to structured parsing and format reconstruction to be converted into a target document, so that the target document retains the complete semantics and structured information of the original knowledge document, and the layout information density of the target document is higher than that of the original knowledge document. Render the target document into a target image; The target image is segmented into a sequence of image blocks, and the image blocks are visually encoded using a visual encoding model to generate a visual knowledge vector based on visual encoding. Based on the visual knowledge vectors, an index is constructed to form an external knowledge base for the artificial intelligence (AI) model. After receiving the query text, the query text is converted into an image using the same method as the target document. Then, it is encoded into a visual query vector based on a visual encoding model. Finally, the visual query vector is used to retrieve matching visual knowledge vectors from the external knowledge base to enhance the AI ​​model's response content generation.

2. The method according to claim 1, characterized in that, The structured information retained in the original knowledge document in the target document includes: The original knowledge document contains chapter levels, paragraphs, lists, code, and / or reference information.

3. The method according to claim 1, characterized in that, The original knowledge documents include documents with a mix of images and text; The structured information retained in the original knowledge document in the target document includes: The target document retains the text, images, hyperlinks, corresponding spatial layouts, and / or semantic relationships from the original knowledge document.

4. The method according to claim 3, characterized in that, The target document is a text-formatted document; When reconstructing the original knowledge document into a structured format to convert it into a target document, the images in the original knowledge document are converted into link addresses, and the anchor position of the link address in the target document is determined. When rendering the target document into a target image, the corresponding image data is obtained according to the link address to restore the image, and the link address at the corresponding anchor point is replaced with the restored image content.

5. The method according to claim 2, characterized in that, If the images in the original knowledge document contain text content, the method further includes: Before structured parsing and format reconstruction, optical character recognition (OCR) technology is used to identify the text content from the image, and the text recognition results and the original image are both retained in the target document.

6. The method according to claim 1, characterized in that, Also includes: After converting the original knowledge document into a target document, the text content in the target document is semantically enhanced using an AI model, so that the target image can be rendered and visual vectors can be generated based on the semantically enhanced text content.

7. The method according to claim 1, characterized in that, Also includes: After generating knowledge vectors based on visual encoding, the knowledge vectors are quantified to generate visual tokens, and a bidirectional mapping relationship is established between the visual tokens and the corresponding image blocks at anchor positions in the target document. The visual token is the smallest input unit of the AI ​​model. During the content generation process using the input knowledge tokens, the AI ​​model decodes multiple output tokens into image blocks, and according to the anchor positions corresponding to the multiple output tokens in the bidirectional mapping relationship, the decoded image blocks are spliced ​​together based on the anchor position information. The text content is then identified and output from the spliced ​​image.

8. A method for generating enhanced search results, characterized in that, include: Receive query text information input by the user; The query text information is converted into an image using the same rendering method as in the knowledge base construction phase; After visually encoding the image using a visual encoder, a visual query vector based on the visual encoding is obtained. The visual query vector is used to retrieve matching visual knowledge vectors from an external knowledge base, and the retrieved target visual knowledge vectors are used as context input to the AI ​​model to enhance the AI ​​model's response content generation; the external knowledge base is established by the method of any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, The external knowledge base also stores visual tokens generated after quantifying visual knowledge vectors, as well as a two-way mapping relationship between visual tokens and anchor positions in the target document; the visual token is the smallest input unit of the AI ​​model. After retrieving the target visual knowledge vector, the process also includes: The visual query vector and the visual token corresponding to the target visual knowledge vector are input into the AI ​​model so that the AI ​​model outputs multiple visual tokens. The multiple output tokens of the AI ​​model are decoded into image blocks, and the anchor point position information of the multiple output tokens in the target document is obtained according to the bidirectional mapping relationship. After stitching the decoded image blocks according to the anchor point location information, the text content in the stitched image is identified and output as the generated answer.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1 to 9.

11. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / computer-executable instructions, characterized in that, When the computer program / computer executable instructions are executed by a processor in an electronic device, they implement the steps of the method according to any one of claims 1 to 9.

Images