Knowledge base question and answer method and device and computer readable storage medium

Abstract

The invention relates to a knowledge base question answering method and device and a computer readable storage medium, and belongs to the technical field of artificial intelligence. A query question input by a user is received; the query problem is analyzed, and keywords in the query problem are extracted; retrieving slice data matched with the query problem from a retrieval enhancement generation system on the basis of the text correlation between the keyword and the slice data; wherein a plurality of slice data are pre-stored in the retrieval enhancement generation system, the slice data comprise text slices and image-text slices, and the image-text slices comprise storage paths of images and image descriptions with text context information of the images; and inputting the retrieved slice data into a multi-modal large model, generating an answer related to the query question and an image matched with the answer, and outputting the question answer with the image. According to the method, the capability of an original retrieval enhancement generation system is remarkably improved, and the information requirement of a user in an image-text environment is more comprehensively met.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a knowledge base question-answering method, device, and computer-readable storage medium. Background Art

[0002] Traditional Retrieval-Augmented Generation (RAG) systems are mainly designed to process and generate text information. They use retrieval technology to find information related to user queries from large amounts of text data, and combine them with generation models to produce detailed and accurate answers.

[0003] However, the traditional search enhancement generation method can only search for similar or related texts, and is basically powerless for image search or image-text mixed search. Therefore, how to implement content output with images in the knowledge base question-answering system is a topic worth studying in this technical field. Summary of the invention

[0004] In view of the above analysis, the embodiments of the present invention aim to provide a knowledge base question and answer method, device and computer-readable storage medium to solve the problem that the existing retrieval enhancement generation method can only retrieve relevant text information and cannot meet the information needs of users in more complex environments.

[0005] In a first aspect of the present application, a knowledge base question answering method is provided, comprising:

[0006] Receive query questions input by users;

[0007] Parsing the query question and extracting keywords therein;

[0008] Based on the text relevance between the keywords and the slice data, slice data matching the query question is retrieved from the retrieval enhancement generation system; wherein the retrieval enhancement generation system pre-stores a plurality of slice data, the slice data including text slices and graphic slices, the graphic slices including the storage path of the image and the image description with the text context information of the image;

[0009] The retrieved slice data is input into the multimodal large model, an answer related to the query question and an image matching the answer are generated, and the answer to the question with the image is output.

[0010] Optionally, the generation process of slice data in the retrieval enhancement generation system is:

[0011] Extracting images from knowledge documents in large-scale databases;

[0012] The image is stored in the cloud, a storage path of the image is obtained as original slice data, and a placeholder is reserved in the original slice data;

[0013] Generate image descriptions with textual contextual information about the image;

[0014] The reserved placeholder is replaced with the storage path and image description of the image to obtain updated slice data.

[0015] Optionally, generating an image description with textual context information of the image includes:

[0016] Call the multimodal large model to extract the visual features of the image and generate a basic description of the image;

[0017] Extract textual contextual information from the document containing the image;

[0018] The multimodal large model is called to generate an image description based on the basic description and the text context information.

[0019] Optionally, the calling of the multimodal large model to generate an image description based on the basic description and the text context information includes:

[0020] Determine the correlation between the image visual features and the semantic features of the text context information;

[0021] If the image visual features are not relevant to the text context information or the image lacks the text context information, generating an image description based on the basic description;

[0022] If the text context information is related to the image content, the basic description and the text context information are fused to generate an image description.

[0023] Optionally, the calling of the multimodal large model to generate an image description based on the basic description and the text context information includes:

[0024] Calculate the similarity between the image visual feature and the semantic feature of the text context to obtain a first similarity score;

[0025] Calculate the similarity between the image visual feature and the semantic feature of the text context information to obtain a second similarity score;

[0026] Based on the first similarity score and the second similarity score, it is determined whether the image is more related to the previous text information or the following text information, and an image description is generated.

[0027] Optionally, after generating the image description with the text context information of the image, the method further includes:

[0028] Check the generated image descriptions for irrelevant content;

[0029] If not, the image description is output in the format of the example.

[0030] Optionally, the step of inputting the retrieved slice data into the multimodal macro model to generate an answer related to the query question and an image matching the answer comprises:

[0031] Calculating the relevance of the image in the graphic slice to the answer related to the query question;

[0032] Based on the relevance of the images to the answer related to the query question, selecting the image with the highest relevance to the answer;

[0033] By utilizing the multimodal processing capabilities of the large multimodal model, the generated answers and the selected images are fused, and the integrated data is used as the final output.

[0034] Optionally, after generating an answer related to the query question and an image matching the answer, the method further includes:

[0035] Converting the answer related to the query question and the image matching the answer into a format suitable for a human-computer interaction interface;

[0036] The answer and an image matching the answer are displayed on a human-computer interaction interface.

[0037] In a second aspect of the present application, a knowledge base question and answer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any one of the above-mentioned knowledge base question and answer methods is implemented.

[0038] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the knowledge base question and answer method described above is implemented.

[0039] The knowledge base question-and-answer method provided in the present application receives a query question input by a user; parses the query question and extracts keywords therein; based on the text relevance between the keywords and the slice data, retrieves slice data matching the query question from a retrieval enhancement generation system; wherein the retrieval enhancement generation system pre-stores a plurality of slice data, the slice data includes text slices and graphic slices, the graphic slices include the storage path of the image and the image description with the text context information of the image; the retrieved slice data is input into a multimodal large model, an answer related to the query question and an image matching the answer are generated, and an answer to the question with an image is output. The present application utilizes the recognition capability of a multimodal large model to realize a highly relevant graphic-text hybrid retrieval enhancement, which can significantly enhance the capability of the original retrieval enhancement generation system, enabling it to effectively process and generate graphic-text hybrid content, thereby more comprehensively meeting the information needs of users in a graphic-text environment. In addition, the present application also provides a knowledge base question-and-answer device and a computer-readable storage medium having the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0041] Figure 1 A flowchart of a specific implementation method of the knowledge base question-answering method provided in this application;

[0042] Figure 2 A schematic diagram of the generation process of slice data in the retrieval enhancement generation system for this application;

[0043] Figure 3 A flowchart of another specific implementation of the knowledge base question-answering method provided in this application;

[0044] Figure 4 This is a structural block diagram of the knowledge base question and answer device provided in this application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0046] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0047] A flowchart of a specific implementation of the knowledge base question-answering method provided in this application is as follows Figure 1 As shown, this embodiment specifically includes:

[0048] S101: receiving a query question input by a user.

[0049] The user can input the query question through the human-computer interaction interface, including text, image or a combination of the two. The human-computer interaction interface can be a web page, mobile application or other forms of interaction interface, which does not affect the implementation of this application.

[0050] S102: Parse the query question and extract keywords therein.

[0051] Parse the query input by the user, which may include operations such as word segmentation, part-of-speech tagging, and named entity recognition, to accurately extract the keywords in the query. Identify the keywords in the query for use in the subsequent retrieval process.

[0052] S103: Based on the text relevance between the keyword and the slice data, the slice data matching the query question is retrieved from the retrieval enhancement generation system.

[0053] The retrieval enhancement generation system pre-stores a plurality of slice data, the slice data including text slices and graphic slices, the graphic slices including the storage path of the image and the image description with the text context information of the image. It is understandable that the text slices include pure text content, while the graphic slices include the storage path of the image and the image description with the text context information of the image.

[0054] Based on the extracted keywords, the retrieval enhancement generation system searches for slice data whose text is highly relevant to the keywords. This includes using a text similarity algorithm to find the most relevant slices, such as using cosine similarity, TF-IDF, and other methods to find the most relevant slices. It is understandable that for image and text slices, the text relevance calculation is performed between the keywords and the image description stored in the slice data to find matching image and text slices.

[0055] like Figure 2 The schematic diagram of the generation process of slice data in the retrieval enhancement generation system is shown as follows:

[0056] S1031: Extract images from knowledge documents in a large-scale database.

[0057] The knowledge documents stored in the database are obtained by using query statements or API calls. According to the format and structure of the knowledge documents, scripts or programs can be used to automatically identify and extract images in the knowledge documents for further processing.

[0058] S1032: Storing the image in the cloud, obtaining the storage path of the image as original slice data, and reserving a placeholder in the original slice data.

[0059] Select a cloud storage service and upload the extracted images to the selected cloud storage service. After uploading, each image will get a unique URL, which will be used as the storage path of the image. The storage path will be used as the original slice data, and a placeholder will be reserved in the original slice data.

[0060] S1033: Generate an image description with textual context information of the image.

[0061] The multimodal large model is called to extract the visual features of the image and generate a basic description of the image; the text context information in the document where the image is located is extracted; and the multimodal large model is called to generate an image description based on the basic description and the text context information.

[0062] Determine the correlation between the image visual features and the semantic features of the text context information; if the image visual features are not correlated with the text context information or the image lacks the text context information, generate an image description based on the basic description; if the text context information is related to the image content, fuse the basic description and the text context information to generate an image description.

[0063] The multimodal large model is called to extract the image visual features from the image. The natural language processing technology is used to extract semantic features from the text context associated with the image. These features represent the meaning and information of the text. The correlation between the image visual features and the semantic features of the text is determined by comparing the two. Specifically, the similarity between the feature vectors can be calculated, such as using cosine similarity and other methods to determine the correlation between the two.

[0064] If the judgment result shows that the image visual features are not relevant to the textual context information, or there is not enough textual context information around the image, then the image description will be generated based on the basic description of the image. This basic description mainly focuses on describing the content that is directly visible in the image. If the image content is relevant to the textual context information, the generation of the image description will combine the basic description and the textual context information. This means that the description not only covers the content that is intuitively displayed in the image, but also includes additional information provided in the text, which at least includes the contextual information of the text.

[0065] When it is determined that text and images are highly relevant, multimodal learning models can be specifically used to generate image descriptions that combine textual information and visual information. These models are able to understand the joint representation of image content and related text, and the generated image descriptions are richer and deeper. It is understood that the generated image descriptions can be further processed, including language polishing and adjustments, to ensure the natural flow of the description and the accuracy of the information. This implementation process not only improves the accuracy and relevance of the description, but also ensures that the generated content is semantically closely connected to the image and its contextual information.

[0066] More specifically, it is possible to further subdivide and determine whether the image visual features are more relevant to the previous text or the following text. The process includes: calculating the similarity between the image visual features and the semantic features of the previous text information to obtain a first similarity score; calculating the similarity between the image visual features and the semantic features of the following text information to obtain a second similarity score; based on the first similarity score and the second similarity score, determining whether the image is more relevant to the previous text information or the following text information, and generating an image description.

[0067] As a specific implementation, cosine similarity can be used to calculate the similarity between two sets of features. Cosine similarity measures the degree of similarity between two vectors in direction and is suitable for comparing feature similarity in high-dimensional space. The similarity between the image visual feature vector and the semantic feature vector of the text context information is calculated to obtain a first similarity score; the similarity between the image visual feature vector and the semantic feature vector of the text context information is calculated to obtain a second similarity score. The first similarity score is compared with the second similarity score. This comparison result will show whether the image content is more relevant to its context or context information. If the first similarity score is higher than the second similarity score, it is considered that the image is more relevant to the text context information; otherwise, it is considered that the image is more relevant to the text context information. Based on the obtained similarity analysis results, the text content that is more relevant to the image (i.e., the text context or the text context) is selected as the main reference for generating the description to generate the image description.

[0068] This process uses a multimodal generative model to generate a detailed image description based on the selected text content and the visual information of the image itself. The image description will combine the visual content and the most relevant text information to provide a comprehensive explanation of the image content, providing more and more comprehensive information for the retrieval enhancement system and providing a basis for the knowledge base question answering system to generate pictures and text answers.

[0069] After generating the image description with the text context information of the image, the method further includes: checking whether the generated image description contains irrelevant content; if not, outputting the image description according to the example format.

[0070] The generated image description may be further manually or automatically checked to determine whether it contains content that is irrelevant to the image. The process of automated checking may be: analyzing keywords and phrases in the image description, and checking whether the relevance of these contents to the image visual features and text context exceeds a threshold, which may be set in advance. In a specific implementation, specific rules may be set or a machine learning model may be trained to automatically identify and mark possible irrelevant content. Once it is confirmed that the generated description does not contain irrelevant content, it may be output according to the format requirements of the example. The format of the example may specifically include a specific text layout, font, included elements, etc. The image description may be generated into a suitable form according to different application scenarios, such as a web page, report, social media post, or other specific form.

[0071] S1034: The reserved placeholder is replaced by the storage path and image description of the image to obtain updated slice data.

[0072] The reserved placeholder can be used for subsequent replacement. After obtaining the image storage path and image description, the placeholder is replaced with the image storage path and the generated image description. The updated slice data is data including text information and image information. This information not only obtains text answers, but also allows you to see highly related images, which can provide more layers and more specific information for the retrieval enhancement system.

[0073] S104: Input the retrieved slice data into the multimodal large model to generate an answer related to the query question and an image matching the answer.

[0074] Calculate the correlation between the image in the graphic slice and the answer related to the query question; based on the correlation between the image and the answer related to the query question, select the image with the highest correlation to the answer; utilize the multimodal processing capability of the multimodal large model to fuse the generated answer and the selected image, and use the integrated data as the final output.

[0075] Based on the multimodal large model, the correlation score between the text features of the query and the visual features of each image is calculated. Based on the calculated correlation score, the image most relevant to the query is selected. This selection process ensures that the selected image best matches the user query in terms of visual content.

[0076] The selected image is merged with the generated text answer. This step uses the power of the multimodal large model to not only integrate text and image data, but also adjust the format and style so that the final output is both informative and visually appealing. The integrated data (including text and images) is the final output in the form of a report, web page display, infographic, or other digital media. The format and style of the output can also be customized according to the application scenario and user needs, which does not affect the implementation of this application.

[0077] This application utilizes the recognition capabilities of a large multimodal model to achieve highly relevant mixed image and text retrieval enhancement, which can significantly improve the capabilities of the original retrieval enhancement generation system, enabling it to effectively process and generate mixed image and text content, thereby more comprehensively meeting users' information needs in a image and text environment.

[0078] The method adopted in this application can generate highly relevant image descriptions at low cost. Compared with directly using a large multimodal model, the image description generated by this application is shorter and can well summarize most of the information of the image, and the generated image description is highly relevant to the contextual text information. In addition, the method of directly using a large multimodal model to generate image descriptions will result in longer corresponding image content, and whether it is embedded or base64, it will take up more slice length, which is much longer than the slice length of the image description generated by the method adopted in this application, and takes up too many hardware resources.

[0079] A flowchart of another specific implementation of the knowledge base question-answering method provided in this application is as follows: Figure 3 As shown, this embodiment specifically includes:

[0080] 301: Document parsing process.

[0081] Get documents in various formats, such as word and pdf.

[0082] As a specific implementation, all document data may be uniformly processed into documents in word format (e.g., converting PDF format into word format). This can simplify subsequent processing steps, because the text content and images in the word format can be extracted relatively easily.

[0083] Parse documents to extract textual content and embedded images in knowledge documents.

[0084] 302: Text slice generation process.

[0085] Splitting long text documents into smaller paragraphs or sentences that are easier to process helps improve the accuracy of retrieval and the relevance of generated content. Specifically, text slices can be made based on structure and content logic, and each text slice contains a set of closely related information, which is convenient for separate processing. Each slice is indexed so that relevant text can be quickly located in subsequent retrieval.

[0086] 303: Graphics and text slice generation process.

[0087] Extract the images in the document and generate detailed and accurate image descriptions in the form of text to describe the embedded images in the document. These descriptions help provide context in subsequent content retrieval and understanding. This part is the core of the implementation of this application.

[0088] Specifically, it includes: using multimodal models to extract image visual features. Based on image visual features and text context (such as the context text of the document where the image is located), generate image descriptions. Generate labels for images to enhance image search capabilities and classification accuracy.

[0089] The implementation process of this step includes the following details:

[0090] Get all the images in the word document through Python code; upload each image to the cloud separately, and get the image link stored in the cloud; insert a placeholder for the image link at the image position of the original document for subsequent replacement.

[0091] The multimodal large model GLM-4V is called to extract the visual features of the image and generate a basic description of the image. The basic description of the image is based on the visual content of the image and provides a preliminary understanding of the image.

[0092] Insert the basic description of the image as a placeholder after the image link corresponding to the original slice data.

[0093] The GLM-4 model is called to infer the image content and title of the new slice based on the context of the image and the basic description of the image, and obtain the image description with context.

[0094] The image link uploaded to the cloud and the image description with context information are used to replace the slice placeholder to obtain the updated slice data.

[0095] The updated slice data now includes links to images with detailed descriptions that are based not only on the image itself but also incorporate the specific context of the document in which the image is found.

[0096] 304: Knowledge base question answering system execution process.

[0097] The user submits a query containing text and / or images through the interface. The RAG system retrieves the most relevant text and image content in the index library based on the user's query and the generated prompt words. Based on the retrieved results, it generates detailed mixed text and image content to respond to the user.

[0098] Receive user input query questions, which may include pure text query, image query or mixed text and image query. Parse the query input by the user to determine the query type and key information. Based on the system knowledge base and existing data, it can also include the process of optimizing query vocabulary to improve the relevance and accuracy of retrieval. Input the optimized query into the RAG system to retrieve relevant slice data and output the answer with image link.

[0099] Based on any of the above embodiments, after generating the answer related to the query question and the image matching the answer, it also includes: converting the answer related to the query question and the image matching the answer into a format suitable for a human-computer interaction interface; and displaying the answer and the image matching the answer on the human-computer interaction interface.

[0100] The embodiment of the present application can greatly improve the ability to process mixed text and image documents through the RAG system that combines multimodal recognition and retrieval enhancement technology, and achieve more efficient and accurate information retrieval and generation. This system is particularly suitable for application scenarios that require a high degree of integration and understanding of multiple data types.

[0101] In addition, the present application also provides a knowledge base question-answering device, such as Figure 4As shown in the structural block diagram of the knowledge base question and answer device provided in the present application, the device specifically includes a memory 41 and a processor 42, and the memory 41 stores a computer program, and when the computer program is executed by the processor 42, any one of the above-mentioned knowledge base question and answer methods is implemented.

[0102] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned knowledge base question and answer methods is implemented.

[0103] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0104] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0105] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0106] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A knowledge base question answering method, characterized in that: include: Receive query questions input by users; Parsing the query question and extracting keywords therein; Based on the text relevance between the keywords and the slice data, slice data matching the query question is retrieved from the retrieval enhancement generation system; wherein the retrieval enhancement generation system pre-stores a plurality of slice data, the slice data including text slices and graphic slices, the graphic slices including the storage path of the image and the image description with the text context information of the image; The retrieved slice data is input into the multimodal large model, an answer related to the query question and an image matching the answer are generated, and the answer to the question with the image is output.

2. The knowledge base question answering method according to claim 1, characterized in that: The generation process of slice data in the retrieval enhancement generation system is as follows: Extracting images from knowledge documents in large-scale databases; The image is stored in the cloud, a storage path of the image is obtained as original slice data, and a placeholder is reserved in the original slice data; Generate image descriptions with textual contextual information about the image; The reserved placeholder is replaced with the storage path and image description of the image to obtain updated slice data.

3. The knowledge base question answering method according to claim 2, characterized in that: The generating of an image description with textual context information of the image comprises: Call the multimodal large model to extract the visual features of the image and generate a basic description of the image; Extract textual contextual information from the document containing the image; The multimodal large model is called to generate an image description based on the basic description and the text context information.

4. The knowledge base question answering method according to claim 3, characterized in that: The calling of the multimodal large model to generate an image description based on the basic description and the text context information includes: Determine the correlation between the image visual features and the semantic features of the text context information; If the image visual features are not relevant to the text context information or the image lacks the text context information, generating an image description based on the basic description; If the text context information is related to the image content, the basic description and the text context information are fused to generate an image description.

5. The knowledge base question answering method according to claim 4, characterized in that: The calling of the multimodal large model to generate an image description based on the basic description and the text context information includes: Calculate the similarity between the image visual feature and the semantic feature of the text context to obtain a first similarity score; Calculate the similarity between the image visual feature and the semantic feature of the text context information to obtain a second similarity score; Based on the first similarity score and the second similarity score, it is determined whether the image is more related to the previous text information or the following text information, and an image description is generated.

6. The knowledge base question answering method according to claim 2, characterized in that: After generating the image description with the textual context information of the image, the method further includes: Check the generated image descriptions for irrelevant content; If not, the image description is output in the format of the example.

7. The knowledge base question answering method according to any one of claims 1 to 6, characterized in that: The step of inputting the retrieved slice data into the multimodal macro model to generate an answer related to the query question and an image matching the answer comprises: Calculating the relevance of the image in the graphic slice to the answer related to the query question; Based on the relevance of the images to the answer related to the query question, selecting the image with the highest relevance to the answer; By utilizing the multimodal processing capabilities of the large multimodal model, the generated answers and the selected images are fused, and the integrated data is used as the final output.

8. The knowledge base question answering method according to claim 7, characterized in that: After generating an answer related to the query question and an image matching the answer, the method further includes: Converting the answer related to the query question and the image matching the answer into a format suitable for a human-computer interaction interface; The answer and an image matching the answer are displayed on a human-computer interaction interface.

9. A knowledge base question-answering device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the knowledge base question and answer method according to any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the knowledge base question and answer method according to any one of claims 1-8 is implemented.

Images