Image-text mixed document question and answer method and system based on fusion of large model and knowledge base

Abstract

The invention belongs to the technical field of artificial intelligence, and provides an image-text mixed document question and answer method and system based on fusion of a large model and a knowledge base, and the method comprises the steps: obtaining a to-be-processed image-text mixed document and a user question; carrying out image, text and table analysis on the obtained image-text mixed document, and respectively constructing a vector knowledge base and a graph knowledge base in combination with the large model; searching user questions in the question knowledge base, when the similarity of the user questions exceeds a similarity threshold value, answering the user questions based on the question knowledge base, otherwise, entering search answering, and completing question answering of the image-text mixed document; and in the retrieval answer, retrieving the user question based on the constructed vector knowledge base and the graph knowledge base, fusing retrieval results of the two knowledge bases to realize association of an image and a text in the image-text mixed document, and obtaining a user question answer with an image identifier based on a large model. And efficient understanding and accurate questioning and answering of the image-text mixed content are realized.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a method and system for question-answering mixed text and image documents based on the fusion of a large model and a knowledge base. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Hybrid document question answering (Q&A) is an intelligent, comprehensive system capable of processing complex documents containing text, images, tables, and charts, providing comprehensive answers based on user input. With the rapid development of artificial intelligence (AI), large models, as pre-trained models with powerful language processing capabilities, can expand application scenarios and improve overall Q&A performance in hybrid document question answering.

[0004] Hybrid document question and answering with text and images can be used in multiple fields, including healthcare, education and training, and business intelligence, to obtain comprehensive and accurate answers by simultaneously referring to text and image information. For example, in medical diagnosis, doctors need to refer to both medical records and imaging images; in education, students need to understand both textbook text and schematic diagrams; and in business analysis, analysts need to combine report text and statistical charts. These scenarios require hybrid document question and answering to be able to simultaneously understand and integrate text and image information and provide accurate answers based on comprehensive knowledge.

[0005] Existing mixed-text and image document question answering has the following processing difficulties: (1) When multimodal elements such as text description images, image supporting text, and table data in a mixed-text document are cited in the main text, it is necessary to truly understand the semantic associations and spatial / logical relationships between different modal elements in order to accurately respond to mixed-text document questions and answers. Otherwise, the mixed-text document question and answer may be confusing. (2) When a mixed-text document contains a complex two-dimensional spatial layout, it is necessary to understand the spatial position relationship of the elements (two-dimensional spatial layout) and the meaning of the image in the context of the mixed-text document in order to accurately position the elements and images; (3) When mixed text and image documents contain charts, effective architecture and training objectives need to be designed to achieve cross-modal information alignment (for example, how to accurately associate "as shown in the figure" in the text with the corresponding chart, and understand the correspondence between specific data points in the chart and the text description).

[0006] At present, large multimodal models (such as GPT-4V, Gemini, Claude 3, LLaVA, Qwen-VL, UniDoc) learn image-text pairing data in the pre-training stage and have preliminary cross-modal understanding capabilities. Large multimodal models can solve the above-mentioned processing difficulties in question-answering mixed documents with images and text. However, cross-modal support for deep semantic fusion and complex logical reasoning remains a research difficulty, and large multimodal models often have image-text separation or association errors. The effect of question-answering mixed documents with images and text with logical associations needs to be improved. Summary of the Invention

[0007] To solve the above problems, the present invention proposes a question-answering method and system for mixed text and image documents based on the fusion of a large model and a knowledge base. By integrating a multimodal large model and multiple knowledge bases, the mixed text and image documents can be effectively processed to achieve efficient understanding and accurate question-answering of mixed text and image content.

[0008] According to some embodiments, a first solution of the present invention provides a question-answering method for hybrid document with images and text based on the fusion of a large model and a knowledge base, which adopts the following technical solutions: A hybrid document question-answering method based on the fusion of a large model and a knowledge base, including: Obtain mixed text and image documents and user questions to be processed; Parse the images, texts, and tables of the acquired mixed-text and image documents, and build a vector knowledge base and a graph knowledge base based on the large model. Retrieve user questions from the preset question knowledge base. When the similarity of the user questions exceeds the similarity threshold, answer the user questions based on the question knowledge base. Otherwise, enter the retrieval answer to complete the question and answer of the mixed text and image documents. In the retrieval answer, user questions are retrieved based on the constructed vector knowledge base and graph knowledge base, and the retrieval results of the two knowledge bases are integrated to realize the association between images and texts in mixed text and image documents, and the user question answers with image labels based on the big model are obtained, completing the mixed text and image document question and answer based on the fusion of the big model and the knowledge base.

[0009] As a further technical limitation, the obtained mixed text and image document to be processed is subjected to structured analysis, and the text, tables and images in the mixed text and image document are subjected to element analysis in the order of the mixed text and image document to obtain the analysis results that filter out the images, texts and tables, and the logical order of arrangement of each element is retained in the obtained analysis results; the analysis results of the obtained text and table are defined as character knowledge content, and the analysis results of the obtained image are subjected to image identification and then injected into the character knowledge content, and the obtained character knowledge content is the vector knowledge base.

[0010] It should be noted that this embodiment uses optical character recognition (OCR) technology to parse the text element information in the mixed text and image document, and uses character mapping to create a dynamically adjustable canvas to parse and identify the table element information in the mixed text and image document; in the process of parsing the image element information, the text element information and table element information in the mixed text and image document are first screened out to obtain the image element information, and then the OCR technology is used to identify the text information in the image, complete the image identification, and inject the identified text information in the image into the character knowledge content.

[0011] Furthermore, based on the multimodal large model, the images in the mixed text and image documents are interpreted, and the image attribute data is obtained by combining the text content containing the image identifier, so as to understand the context of the image in the document and improve the association between the image and the document content; each image identifier in the mixed text and image document is regarded as an entity node, and the graph structure of each entity node is constructed to complete the construction of the graph knowledge base.

[0012] Furthermore, the obtained parsing results are sliced, and the entities and relationships of each slice are extracted based on the language big model. The big model is used to generate descriptions of the entities and relationships, and the multimodal big model is used to generate image entity descriptions; the obtained descriptions are generated and stored in the graph knowledge base.

[0013] As a further technical limitation, the preset question knowledge base includes several answers and several questions; wherein the answers are in a multimodal form including at least text, table and image, and one answer may correspond to several questions.

[0014] As a further technical limitation, vectorization operations are performed on user questions based on a preset question knowledge base, user questions are retrieved in the question knowledge base, and question vectors of user questions are obtained. By comparing the similarity between the user questions and the obtained question vectors of user questions, the answer method of the user questions is determined.

[0015] As a further technical limitation, when the obtained similarity exceeds a similarity threshold, an answer matching the user question is searched in the question knowledge base, and the question and answer of the mixed text and image document is completed according to the matched answer.

[0016] As a further technical limitation, when the similarity obtained does not exceed the similarity threshold, the question and answer of the mixed text and image document is completed by retrieving the answer; the key question and answer entities and relationships of the user questions are extracted, and the extracted results are searched for similarity in the vector knowledge base. The search scope is the entity name, description and relationship description in the graph knowledge base. After the search, the previous nThe most similar entities and relationships are found; the acquired entities and relationships are traversed in the graph knowledge base to obtain entity association information, the obtained entity association information is summarized and converted into a text description, and the associated content of the object entity in the user question is integrated based on the text description.

[0017] Furthermore, the acquired entities and relationships are sliced, and the sliced ​​entities and relationships are combined with the top n slices with high similarity to the user question and deduplicated to obtain the slice text associated with the user question, and all the slice texts are summarized; the obtained text description and slice text are understood by the large model to complete the retrieval answer and obtain the answer to the user question.

[0018] According to some embodiments, the second solution of the present invention provides a hybrid document question-answering system based on the fusion of a large model and a knowledge base, which adopts the following technical solutions: A hybrid document question-answering system based on the fusion of large models and knowledge bases, including: An acquisition module configured to acquire a mixed text-image document and a user question to be processed; A construction module is configured to parse the images, texts, and tables of the acquired mixed-text and image documents, and to construct a vector knowledge base and a graph knowledge base respectively in combination with the large model; The question-answering module is configured to retrieve user questions from a preset question knowledge base. When the similarity of the user questions exceeds a similarity threshold, the user questions are answered based on the question knowledge base. Otherwise, the module searches for answers to complete the question-answering of mixed text and image documents. In the retrieval answer, user questions are retrieved based on the constructed vector knowledge base and graph knowledge base, and the retrieval results of the two knowledge bases are integrated to realize the association between images and texts in mixed text and image documents, and the user question answers with image labels based on the big model are obtained, completing the mixed text and image document question and answer based on the fusion of the big model and the knowledge base.

[0019] Compared with the prior art, the present invention has the following beneficial effects: When answering questions in mixed text and image documents, the present invention provides answers in a multimodal form, rather than parsing the image content and converting the parsing results into text descriptions. When asking and answering questions, the text answer to the question and its reference image are obtained at the same time, so as to avoid inaccurate answers to questions due to deviations in the understanding of image content; the vector knowledge base and the graph knowledge base are integrated to effectively process mixed text and image documents, realizing efficient understanding of mixed text and image content and accurate question and answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting a part of the specification of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions of this embodiment are used to explain this embodiment and do not constitute an improper limitation on this embodiment.

[0021] Figure 1 This is a flowchart of the image-text hybrid document question-answering method based on the fusion of a large model and a knowledge base in Example 1 of the present invention; Figure 2 This is an architecture diagram of the graph knowledge base in the first embodiment of the present invention; Figure 3 Schematic diagram of the data structure in the graph knowledge base in the first embodiment of the present invention; Figure 4 This is a flowchart of the search and answer phase in the first embodiment of the present invention; Figure 5 This is a structural block diagram of the graphic-text hybrid document question-answering system based on the fusion of a large model and a knowledge base in Example 2 of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0024] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0025] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention, and should not be understood as limiting the present invention.

[0026] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0027] Example 1 The first embodiment of the present invention introduces a hybrid document question-answering method based on the fusion of a large model and a knowledge base.

[0028] like Figure 1 The illustrated method for question answering hybrid documents based on a large model and knowledge base integration includes: Obtain mixed text and image documents and user questions to be processed; Parse the images, texts, and tables of the acquired mixed-text and image documents, and build a vector knowledge base and a graph knowledge base based on the large model. Retrieve user questions from the preset question knowledge base. When the similarity of the user questions exceeds the similarity threshold, answer the user questions based on the question knowledge base. Otherwise, enter the retrieval answer to complete the question and answer of the mixed text and image documents. In the retrieval answer, user questions are retrieved based on the constructed vector knowledge base and graph knowledge base, and the retrieval results of the two knowledge bases are integrated to realize the association between images and texts in mixed text and image documents, and the user question answers with image labels based on the big model are obtained, completing the mixed text and image document question and answer based on the fusion of the big model and the knowledge base.

[0029] It should be noted that the similarity threshold can be adaptively set according to the mixed text-image document and the question knowledge base.

[0030] It should be noted that the mixed text and image document in this embodiment includes at least text, images and tables.

[0031] As one or more implementation methods, the obtained mixed text and image document is subjected to structured analysis, and the text, tables and images in the mixed text and image document are subjected to element analysis according to the order in the mixed text and image document to obtain the analysis results that filter out the images, texts and tables, and the layout analysis technology is used to retain the logical order of the arrangement of each element in the obtained analysis results; the obtained text and table analysis results are defined as character knowledge content, and the obtained image analysis results are subjected to image identification and then injected into the character knowledge content, and the obtained character knowledge content is the vector knowledge base.

[0032] It should be noted that this embodiment slices the results obtained from the analysis, embeds the slice content into words (semantic vectorization), and stores it in the vector knowledge base; the character content of the slice enters the knowledge graph module in subsequent steps to extract entities and relationships about the picture and related content, and construct a knowledge graph.

[0033] This embodiment stores the filtered image analysis results and constructs an image repository; at the same time, the constructed image repository records relevant data corresponding to the image, such as the image identifier and the source of the mixed image-text document.

[0034] As one or more embodiments, Figure 2As shown, a multimodal large model is used to understand the images in the mixed text-image document to realize the interpretation of the content of the image itself in the mixed text-image document; the image attribute data is obtained by combining the text content containing the image identifier, the content of the image in the context of the document is understood, and the relationship between the image and the document content is improved. Each image identifier in the mixed text-image document is taken as an entity node, and a graph structure of each entity node is constructed, and the constructed graph structure is stored in the graph knowledge base; the language large model is used to extract entities and relationships from each slice of the document (entity extraction includes extraction of image entities and extraction of other entities associated with the image) to establish the relationship between the image and other entities in the text description; the large model is then combined to generate descriptions of entities and relationships, and the multimodal large model is used to generate descriptions of image entities, and the obtained descriptions are stored in the graph knowledge base, and the image entities in the mixed text-image document are recorded with image identifiers. Combined Figure 3 In the data of the graph knowledge base shown, each entity and relationship is associated with the corresponding slice for extraction. At the same time, the name and description of the entity, the relationship and description are vectorized and stored in the vector knowledge base.

[0035] It should be noted that in the process of constructing the graph knowledge base, knowledge graph technology is used. The specific construction process belongs to the existing technology that technical personnel in this field should know, and this embodiment will not be repeated here.

[0036] It should be noted that the question knowledge base preset in this embodiment can support self-entry of questions and answers, wherein the entered answers support multimodal forms including text, tables, images and other data. At the same time, one answer can correspond to multiple questions, and these questions can be entered into the question knowledge base and vectorized for retrieval during subsequent question and answering.

[0037] In this embodiment, the answers entered into the question knowledge base at least include character knowledge content and image identifiers.

[0038] When a user asks a question in this embodiment, a search is performed in a preset question knowledge base, a vectorization operation of the user question is performed based on the preset question knowledge base, the user question is retrieved in the question knowledge base, and a question vector of the user question is obtained. By comparing the similarity between the user question and the obtained question vector of the user question, the answer method of the user question is determined.

[0039] As one or more implementation methods, when the obtained user question similarity exceeds the similarity threshold, it can be considered that the user question appears in the question knowledge base. First, find a question that is adapted to the user question in the question knowledge base, and then find an answer that matches the adapted question in the question knowledge base based on the adapted question found. Then, complete the question and answer of the mixed text and image document based on the matched answer.

[0040] It should be noted that similarity is a measure of the degree of similarity between two objects. Depending on the application scenario and data type, similarity can be calculated through similarity calculations based on numerical data, including Euclidean distance, Manhattan distance, Chebyshev distance, and Mahalanobis distance; similarity calculations based on vector direction, including cosine similarity and Pearson correlation coefficient; and similarity calculations based on sets, including Jaccard correlation coefficient and Jaccard distance.

[0041] In this embodiment, similarity is calculated based on the similarity of the vector direction. In other embodiments, similarity can be calculated using other methods. The similarity calculation process belongs to the existing technology that those skilled in the art should know, and this embodiment will not be repeated here.

[0042] As one or more implementation methods, when the similarity of the obtained user questions does not exceed the similarity threshold, the question and answer of the mixed text and image document is completed by retrieving the answer. The specific process is as follows: Figure 4 As shown: Key question-answer entities and relationships are extracted from the user's question. The extracted results are then subjected to a similarity search within the vector knowledge base, targeting the names and descriptions of the entities, relationships, and descriptions stored in the knowledge graph module. The top n most similar entities and relationships are retrieved after the search. For each entity and relationship, graph retrieval is used to retrieve the corresponding data stored in the graph knowledge base. For each entity and relationship, the graph is traversed to identify other entities and relationships associated with the entity or relationship, obtaining their names, descriptions, identifiers, and other information. All associated information is then aggregated and converted into text descriptions, thereby integrating the content associated with the entity inquired about in the user's question. The slices corresponding to the relevant entities and relationships are also recorded and combined with the top n slices with the highest similarity to the user's question, eliminating duplication and summarizing the text content of the slices associated with the user's question. Finally, the resulting slice text content and text description are fed into the large model for understanding, completing the retrieval and answering, ultimately providing the answer to the user's question.

[0043] It should be noted that this embodiment searches from the graph database and the vector knowledge base respectively, and realizes the effective association between the image and the document text content by combining the two knowledge retrieval results. Through the prompt word engineering, the large model is used to give the original answer with the image identifier in the answer; when it is detected that the original answer contains an image identifier, it is replaced with the corresponding image data in the image repository and rendered, thereby realizing the multimodal display of images and texts in the answer.

[0044] This embodiment provides answers in a multimodal form when answering mixed text and image documents, rather than parsing the image content and converting the parsing results into text descriptions. When asking and answering questions, the text answer to the question and its reference image are obtained at the same time to avoid inaccurate answers to questions due to deviations in the understanding of image content. The vector knowledge base and the graph knowledge base are integrated to effectively process mixed text and image documents, achieving efficient understanding of mixed text and image content and accurate question and answer.

[0045] Example 2 The second embodiment of the present invention introduces a hybrid document question-answering system based on the fusion of a large model and a knowledge base.

[0046] like Figure 5 The system is a hybrid document question-answering system based on a large model and knowledge base, including a document processing module, a file storage module, a knowledge graph module, a question-answering module, and a question-answering interaction module. The system completes the entire process from information extraction to answer generation through collaboration between these modules. Specifically: (1) Document processing module The document processing module is responsible for performing structural analysis on the input document, parsing the text, tables, images, figure titles and other elements in the mixed text and image document in sequence, and using layout analysis technology to retain their logical order of arrangement; the images in the mixed text and image document will be converted into image identifiers and injected into the character content.

[0047] It should be noted that the hybrid document question-answering system for images and text, based on the fusion of a large model and a knowledge base, in this embodiment slices the parsed results, embeds the sliced ​​content into words (semantic vectorization), and stores it in the vector knowledge base. Simultaneously, the sliced ​​character content enters the knowledge graph module for entity and relationship extraction related to the image and related content, as well as knowledge graph construction.

[0048] (2) File storage module The file storage module stores the image data extracted from the document and records relevant data corresponding to the image, such as the image identifier and the source document.

[0049] (3) Knowledge graph module The knowledge graph module in this embodiment depicts the attributes and relationships of the image, and uses a multimodal large model to understand the image and interpret the content of the image itself; the text content with the image identifier will also participate in the establishment of the image attributes, and further improve its relationship by understanding the document context content in which the image is located. Each image identifier is regarded as an entity node, and its graph structure is established, and the graph structure is stored in the graph knowledge base; the language large model is then used to extract entities and relationships from each slice of the document. The entity extraction in this embodiment includes the extraction of image entities and other entities associated with the image to establish the relationship between the image and other entities in the text description, and then the large model is used to describe and generate entities and relationships. The description of the image entity is generated using the multimodal large model, and the relevant results are stored in the graph knowledge base. The image identifier will also be recorded for the image entity; each entity and relationship will also be associated with the corresponding slice for extraction, and the name and description of the entity, the relationship and description will also be vectorized and stored in the vector knowledge base.

[0050] (4) Question-Answer Module The question-and-answer module in this embodiment supports self-entry of questions and answers. The answers support multimodal forms, that is, text, tables, images and other data can be entered; one answer can correspond to multiple questions, and these questions will enter the question knowledge base and be vectorized for retrieval during subsequent question-and-answer sessions.

[0051] (5) Question and answer interaction module When a user asks a question, a search is performed in the question knowledge base. If the question has a high similarity with the question vector (the similarity is higher than the preset similarity threshold), the answer corresponding to the question is directly returned as the answer to the user's question. Otherwise, the search and answer stage is entered.

[0052] During the retrieval and answering phase, key question-answering entities and relationships are extracted based on user questions. The extracted results are then searched for similarity in the vector knowledge base. The search scope is the names and descriptions of the entities, relationships, and descriptions stored in the knowledge graph module. After the search, the top n most similar entities and relationships are obtained. For each entity and relationship, graph retrieval is used to obtain the corresponding data stored in the graph knowledge base. For the obtained entities and relationships, the graph is traversed to other entities and relationships associated with the entity or relationship and their names, descriptions, identifiers, and other information are obtained. All associated information is aggregated and converted into text descriptions, thereby integrating the associated content with the entity inquired about in the user's question. At the same time, the slices corresponding to the relevant entities and relationships are also recorded, and combined with the top n slices with the highest similarity to the user's question and deduplicated to obtain the slice text content associated with the user's question. Through the understanding of the large model, the answer is given.

[0053] This embodiment searches from both a graph database and a vector knowledge base, combining the results of these two knowledge retrievals to effectively associate images with document text. Using a large model, a prompt word project is used to present the original answer with an image identifier. In the interactive module frontend, when an image identifier is detected in the original answer, it is replaced with the corresponding image data from the image repository and rendered, thus achieving a multimodal presentation of images and text within the answer.

[0054] The detailed steps are the same as those of the graphic-text hybrid document question-answering method based on the fusion of a large model and a knowledge base provided in Example 1, and will not be repeated here.

[0055] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0056] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0057] The above description is merely a preferred embodiment of this embodiment and is not intended to limit this embodiment. Those skilled in the art will readily appreciate that this embodiment may be modified and varied in various ways. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this embodiment shall be within the scope of protection of this embodiment.

Claims

1. A hybrid document question answering method based on the fusion of large models and knowledge bases, characterized by: include: Obtain mixed text and image documents and user questions to be processed; Parse the images, texts, and tables of the acquired mixed-text and image documents, and build a vector knowledge base and a graph knowledge base based on the large model. Retrieve user questions from the preset question knowledge base. When the similarity of the user questions exceeds the similarity threshold, answer the user questions based on the question knowledge base. Otherwise, enter the retrieval answer to complete the question and answer of the mixed text and image documents. In the retrieval answer, user questions are retrieved based on the constructed vector knowledge base and graph knowledge base, and the retrieval results of the two knowledge bases are integrated to realize the association between images and texts in mixed text and image documents, and the user question answers with image labels based on the big model are obtained, completing the mixed text and image document question and answer based on the fusion of the big model and the knowledge base.

2. A method for question-answering a hybrid document based on a large model and a knowledge base as described in claim 1, characterized in that: Performing structural parsing on the obtained mixed-text and image document to be processed, performing element parsing on the text, tables, and images in the mixed-text and image document according to the order in which they appear in the mixed-text and image document, obtaining a parsing result that filters out the images, text, and tables, and retaining the logical order of arrangement of the elements in the obtained parsing result; The obtained text and table parsing results are defined as character knowledge content, and the obtained image parsing results are subjected to image identification and then injected into the character knowledge content. The obtained character knowledge content is the vector knowledge base.

3. A method for question-answering a hybrid document based on a large model and a knowledge base as described in claim 2, characterized in that: Based on the multimodal large model, images in mixed text and image documents are interpreted, and image attribute data is obtained by combining the text content containing the image identifier, so as to understand the context of the image in the document and improve the association between the image and the document content; each image identifier in the mixed text and image document is regarded as an entity node, and the graph structure of each entity node is constructed to complete the construction of the graph knowledge base.

4. A method for question-answering a hybrid document based on a large model and a knowledge base as described in claim 2, characterized in that: The obtained parsing results are sliced, and the entities and relationships of each slice are extracted based on the language big model. The big model is used to generate descriptions of the entities and relationships, and the multimodal big model is used to generate image entity descriptions; the descriptions generated are stored in the graph knowledge base.

5. The method for question-answering a hybrid document based on a large model and a knowledge base as described in claim 1, characterized in that: The preset question knowledge base includes several answers and several questions; wherein the answers are in a multimodal form including at least text, table and image, and one answer may correspond to several questions.

6. A method for question-answering a hybrid document based on a large model and a knowledge base as described in claim 1, characterized in that: Perform vectorization operations on user questions based on a preset question knowledge base, retrieve user questions from the question knowledge base, obtain question vectors of user questions, and determine the answer method for user questions by comparing the similarity between user questions and the obtained question vectors of user questions.

7. A method for question-answering a hybrid document based on a large model and a knowledge base as described in claim 1, characterized in that: When the obtained similarity exceeds a similarity threshold, an answer matching the user's question is searched in the question knowledge base, and the question and answer of the mixed text-image document is completed according to the matched answer.

8. The method for question-answering a hybrid document of text and images based on the fusion of a large model and a knowledge base as described in claim 1, characterized in that: When the similarity obtained does not exceed the similarity threshold, the question and answer of the mixed text and image document is completed by retrieving the answer; the key question and answer entities and relationships of the user questions are extracted, and the extracted results are searched for similarity in the vector knowledge base. The search scope is the entity name, description and relationship description in the graph knowledge base. After the search, the previous n The most similar entities and relationships are found; the acquired entities and relationships are traversed in the graph knowledge base to obtain entity association information, the obtained entity association information is summarized and converted into a text description, and the associated content of the object entity in the user question is integrated based on the text description.

9. A method for question-answering a hybrid document of text and images based on the fusion of a large model and a knowledge base as described in claim 8, characterized in that: The acquired entities and relationships are sliced, and the sliced ​​entities and relationships are combined with the top n slices with high similarity to the user question and deduplicated to obtain the slice text associated with the user question and summarize all the slice texts; the obtained text description and slice text are understood by the large model to complete the retrieval answer and obtain the answer to the user question.

10. A hybrid document question-answering system based on the fusion of large models and knowledge bases, characterized by: include: An acquisition module configured to acquire a mixed text-image document and a user question to be processed; A construction module is configured to parse the images, texts, and tables of the acquired mixed-text and image documents, and to construct a vector knowledge base and a graph knowledge base respectively in combination with the large model; The question-answering module is configured to retrieve user questions from a preset question knowledge base. When the similarity of the user questions exceeds a similarity threshold, the user questions are answered based on the question knowledge base. Otherwise, the module searches for answers to complete the question-answering of mixed text and image documents. In the retrieval answer, user questions are retrieved based on the constructed vector knowledge base and graph knowledge base, and the retrieval results of the two knowledge bases are integrated to realize the association between images and texts in mixed text and image documents, and the user question answers with image labels based on the big model are obtained, completing the mixed text and image document question and answer based on the fusion of the big model and the knowledge base.

Images