Multi-modal knowledge base management method and device and storage medium

Abstract

The embodiment of the invention provides a management method and device of a multi-modal knowledge base and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: extracting first text content and image data of an original document; analyzing the image data and acquiring second text content corresponding to the image data; combining the first text content and the second text content to obtain mixed text content corresponding to the original document; performing block processing on the mixed text content to obtain a corresponding text content block set; and constructing a multi-modal knowledge base based on the text content block set. According to the embodiment of the invention, the image data of the original document is extracted and converted into the text content, and then the mixed text content based on the combination of the first text content of the original document and the second text content of the graph-to-text is subjected to block processing, so that the efficiency and flexibility of multi-modal data embedding can be effectively improved; therefore, the processing efficiency of the multi-modal knowledge is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a management method, device and storage medium for a multimodal knowledge base. Background Art

[0002] With the advancement of science and technology, LLM (Large Language Model) has been increasingly widely used in the field of natural language processing. Traditional text retrieval technology performs well in processing single text information, but with the growth of multimodal data (such as tables, images, etc.), single text retrieval has been unable to meet actual needs. The emergence of multimodal knowledge retrieval technology can well cope with this challenge. By combining information from different modalities, it effectively improves the accuracy and comprehensiveness of knowledge retrieval.

[0003] However, existing multimodal embedding models need to process a large amount of token data (the basic data unit of LLM processing) when processing image embedding data, which greatly increases the resource consumption of the system and leads to low efficiency of multimodal knowledge processing. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a management method, device and storage medium for a multimodal knowledge base, so as to improve the efficiency of multimodal knowledge processing.

[0005] In a first aspect, an embodiment of the present application provides a method for managing a multimodal knowledge base, including:

[0006] extracting first text content and image data of the original document;

[0007] parsing the image data and acquiring second text content corresponding to the image data;

[0008] Combining the first text content and the second text content to obtain mixed text content corresponding to the original document;

[0009] Processing the mixed text content in blocks to obtain a corresponding text content block set;

[0010] A multimodal knowledge base is constructed based on the text content block set.

[0011] In an embodiment of the present application, by extracting the image data of the original document and converting it into text content, and then performing block processing on the mixed text content based on the combination of the first text content of the original document and the second text content of the image-generated text, the efficiency and flexibility of multimodal data embedding can be effectively improved, thereby improving the processing efficiency of multimodal knowledge.

[0012] In some possible embodiments, after constructing a multimodal knowledge base based on the text content block set, the method further includes:

[0013] In response to a question query request, at least one target text content block matching the question query request is selected from the text content block set;

[0014] An answer result corresponding to the question query request is determined based on the at least one target text content block.

[0015] In an embodiment of the present application, by first screening out text content blocks that match the query question from the overall text content block set, and using this as the data basis for answering the query question, the efficiency and accuracy of multimodal knowledge retrieval can be further improved.

[0016] In some possible embodiments, each text content block in the text content block set is stored in a vector database corresponding to the multimodal knowledge base in the form of a text vector;

[0017] The step of responding to the question query request and selecting at least one target text content block matching the question query request from the text content block set includes:

[0018] Responding to a question query request, and obtaining a query text vector corresponding to the question query request;

[0019] Traversing and calculating the vector similarity between the query text vector and the text vectors of each of the text content blocks;

[0020] Based on the vector similarities corresponding to the text content blocks, a text content block that meets a preset vector similarity condition is used as at least one target text content block that matches the question query request.

[0021] In the embodiment of the present application, the efficiency and flexibility of multimodal knowledge retrieval are further improved by storing text content blocks in the form of vectors and filtering target text content blocks by calculating vector similarity.

[0022] In some possible embodiments, the multimodal knowledge base management method further includes:

[0023] In the case where it is determined that the at least one target text content block contains the second text content, the target image data corresponding to the at least one target text content block is acquired based on a pre-stored image-text mapping relationship; wherein the image-text mapping relationship is a mapping relationship between pre-recorded image data and its corresponding second text content;

[0024] The target image data is output as a reference answer result corresponding to the question query request.

[0025] In an embodiment of the present application, by recording the mapping relationship between the graphic content and the image data, when the screened text content block related to the query question contains graphic content, the corresponding image data is output as the answer result, thereby effectively improving the accuracy and comprehensiveness of multimodal knowledge retrieval.

[0026] In some possible embodiments, the block processing of the mixed text content to obtain a corresponding text content block set includes:

[0027] The first text content in the mixed text content is divided into blocks according to a preset upper limit on the number of characters, and the second text content in the mixed text content is divided into blocks according to unit image data as a whole block to obtain a corresponding text content block set.

[0028] In an embodiment of the present application, by dividing the text content of the original document and the text content of the image-generated text into blocks according to different rules, not only can the integrity of the image-generated text content information be guaranteed as much as possible, but also the contextual connection between the text and the image can be better preserved, thereby further improving the accuracy of multimodal knowledge retrieval.

[0029] In some possible embodiments, parsing the image data and acquiring the second text content corresponding to the image data includes:

[0030] Recognize the text content in the image data based on optical character recognition technology to obtain second text content corresponding to the image data;

[0031] Alternatively, the image data is processed based on a pre-trained visual language model to obtain second text content corresponding to the image data output by the visual language model.

[0032] In the embodiments of the present application, by parsing the text content of image data based on different methods, the flexibility and completeness of extracting image information can be improved, thereby further improving the accuracy of multimodal knowledge retrieval.

[0033] In some possible embodiments, extracting the first text content and image data of the original document includes:

[0034] Extracting first text content and image data of the original document, and recording first position information of the first text content and second position information of the image data;

[0035] The combining the first text content and the second text content to obtain mixed text content corresponding to the original document includes:

[0036] The first text content and the second text content are sorted and combined based on the first position information and the second position information to obtain mixed text content corresponding to the original document.

[0037] In an embodiment of the present application, by recording the location information of the text content and image data of the original document and sorting and combining the converted text content based on the location information, the context information of the original document can be better retained, thereby further improving the accuracy of multimodal knowledge retrieval.

[0038] In a second aspect, an embodiment of the present application provides a management device for a multimodal knowledge base, including:

[0039] A data extraction module, used to extract first text content and image data of an original document;

[0040] An image parsing module, used for parsing the image data and acquiring second text content corresponding to the image data;

[0041] A text combining module, used for combining the first text content and the second text content to obtain mixed text content corresponding to the original document;

[0042] A text block module, used for processing the mixed text content into blocks to obtain a corresponding text content block set;

[0043] The knowledge base construction module is used to construct a multimodal knowledge base based on the text content block set.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the method described in any embodiment of the first aspect when executing the program.

[0045] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented.

[0046] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, the method described in any embodiment of the first aspect can be implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a process for managing a multimodal knowledge base provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of an intelligent question-and-answer application scenario based on a multimodal knowledge base provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a multimodal knowledge base management device provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0053] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0054] It should be noted that when processing image embedding, LLM (Large Language Model) needs to generate and process a large amount of token data, which greatly increases the resource consumption of the system and leads to low efficiency in multimodal knowledge processing.

[0055] In response to the problems existing in the above-mentioned prior art, an embodiment of the present application provides a method for managing a multimodal knowledge base, which reduces the amount of token data to be processed, thereby effectively improving the processing efficiency of multimodal knowledge.

[0056] like Figure 1 As shown, the embodiment of the present application provides a method for managing a multimodal knowledge base, which may include the steps of:

[0057] S1. Extracting first text content and image data of an original document.

[0058] Specifically, the original document is a pre-collected basic document that needs to be extracted. Exemplarily, the open source PyMuPDF tool can be used to parse the incoming original document (usually in a document format such as PDF) to extract the first text content and image data therein. The first text content is the plain text content in the basic document, and the image data is the picture in the basic document.

[0059] S2. Parse the image data and obtain second text content corresponding to the image data.

[0060] The extracted image data can be parsed and the corresponding second text content can be generated. For example, the text content in the image can be recognized by using technologies such as OCR (Optical Character Recognition), and the content and context information of the image can be understood by using VLM (Visual Language Model), and the corresponding second text content can be generated.

[0061] It should be noted that there is a one-to-one correspondence between the image data and the generated second text content, that is, one copy of the second text content is generated for each piece of image data.

[0062] It is understandable that the first text content and the second text content are both text content in the same format. In some embodiments, the difference between the two is that corresponding tags can be added to the first text content and / or the second text content to identify whether the text content is derived from text directly extracted from the original document or text generated from an image, so as to distinguish them in subsequent processing.

[0063] S3. Combine the first text content and the second text content to obtain mixed text content corresponding to the original document.

[0064] Specifically, after extracting text from the original document and parsing image data to generate this text (image-generated text), a mixed text content in a plain text format that is a combination of the first text content and the second text content can be obtained.

[0065] S4. Divide the mixed text content into blocks to obtain a corresponding text content block set.

[0066] Specifically, by dividing the mixed text content into blocks, a set consisting of multiple text content blocks can be obtained. Among them, the mixed text content can be divided into blocks in the form of paragraphs based on technologies such as line break recognition, so that the text content of each paragraph is a separate block; in addition, the text content blocks can be divided based on a preset length (character length or number of sentences), so that the data volume of each text content block remains relatively uniform.

[0067] S5. Build a multimodal knowledge base based on a collection of text content blocks.

[0068] Specifically, based on the divided text content block set, a multimodal knowledge base corresponding to the original document can be constructed. In the subsequent maintenance process, one or more other original documents can be converted into a text content block set in the same form, and the multimodal knowledge base can be updated.

[0069] The embodiment of the present application can effectively improve the efficiency and flexibility of multimodal data embedding by extracting image data of the original document and generating corresponding text content, and then combining the text of the image-generated text with the text extracted from the original document for block processing, thereby improving the processing efficiency of multimodal knowledge.

[0070] In some possible embodiments, after step S5, the following steps may also be included:

[0071] S601, responding to a question query request, and screening out at least one target text content block matching the question query request from a text content block set;

[0072] S602: Determine an answer result corresponding to the question query request based on at least one target text content block.

[0073] like Figure 2 As shown, it should be noted that after building a multimodal knowledge base, functions such as content retrieval and intelligent question and answer can be implemented based on the multimodal knowledge base.

[0074] Specifically, when a question query request input by a user is received, the question query information can be extracted, and then matched with a set of text content blocks based on the extracted question query information, and the matching target text content blocks are filtered out, and then the filtered target text content blocks are used as the data basis for answering the question.

[0075] After filtering the target text content blocks, the pre-trained large language model can be used to perform semantic understanding based on these target text content blocks as context content and output corresponding answer results.

[0076] In some possible embodiments, each text content block in the text content block set is stored in a vector database corresponding to the multimodal knowledge base in the form of a text vector;

[0077] Step S601, responding to a question query request and selecting at least one target text content block matching the question query request from a text content block set, may include:

[0078] S6011, responding to a question query request and obtaining a query text vector corresponding to the question query request;

[0079] S6011, traverse and calculate the vector similarity between the query text vector and the text vector of each text content block;

[0080] S6011. Based on the vector similarities corresponding to the text content blocks, a text content block that meets a preset vector similarity condition is used as at least one target text content block that matches the question query request.

[0081] It should be noted that each text content block can be stored in the form of a text vector in a vector database corresponding to the multimodal knowledge base.

[0082] Exemplarily, after the block processing, the text of each text content block may be vectorized to obtain a corresponding text content block in a vector representation form.

[0083] Exemplarily, before the block processing, the first text content and the second text content (or the combined mixed text content) may be vectorized, and then the vectorized data may be subjected to block processing.

[0084] Then, for the question query request input by the user, vectorization processing is performed in the same way to generate the corresponding query text vector (vector representation of the question query information). Then, the similarity between the query text vector and the vector of each text content block is traversed and calculated; finally, the corresponding text content block whose similarity meets the preset condition is screened as the target text content block. Among them, the preset condition can be a preset similarity threshold. Accordingly, if it is judged that the similarity between the query text vector and the vector of a certain text content block exceeds the similarity threshold, it is determined that the vector similarity of the text content block meets the preset vector similarity condition; in addition, the preset condition can also be the screening number with the highest vector similarity. Accordingly, the n text content blocks with the highest vector similarity (pre-set according to the needs) are screened as the target text content blocks.

[0085] In some possible embodiments, the method for managing a multimodal knowledge base may further include:

[0086] S701, when it is determined that at least one target text content block contains second text content, obtaining target image data corresponding to at least one target text content block based on a pre-stored image-text mapping relationship; wherein the image-text mapping relationship is a mapping relationship between pre-recorded image data and its corresponding second text content;

[0087] S702: Output the target image data as a reference answer result corresponding to the question query request.

[0088] It should be noted that in the process of responding to the user's question query request and processing and outputting the answer result, in addition to outputting the text answer content, the relevant image data can also be output as a reference answer result.

[0089] Specifically, in the process of parsing the image data and generating the second text content, the image-text mapping relationship between the image data and the second text content can be recorded, that is, one image corresponds to one second text content, and the corresponding path information is recorded after the image data is saved.

[0090] Based on this, after filtering out the target text content blocks, it can be determined whether these target text content blocks contain second text content. If so, based on which second text content the current target text content block contains and the pre-recorded image-text mapping relationship, the corresponding target image data and its path information can be determined. Finally, the corresponding target image data can be obtained based on the path information and output as a reference answer result.

[0091] In some possible embodiments, step S4, dividing the mixed text content into blocks to obtain a corresponding set of text content blocks, may include:

[0092] S401. Divide the first text content in the mixed text content into blocks according to a preset upper limit on the number of characters, and divide the second text content in the mixed text content into blocks according to unit image data as a whole block to obtain a corresponding text content block set.

[0093] It should be noted that, in the process of block processing, if the block is divided according to the preset maximum character limit, since the block will be forced to be divided whenever the number of characters reaches the maximum character limit, for the second text content generated by the image, this block method may cause the second text content to lose part of the complete image information, or may mix the content of the image-generated text with the irrelevant first text content, thereby adversely affecting knowledge retrieval. Therefore, in order to improve the reliability of knowledge base construction, different block processing methods should be used for the first text content and the second text content.

[0094] Specifically, for the first text content extracted directly from the original document, it can be processed in blocks according to a preset upper limit on the number of characters; and for the second text content, considering the contextual connection between the image-generated text content and the first text content, if in the order of the original document, the previous text content block corresponding to the current image does not exceed the maximum number of characters, the second text content corresponding to the image can be merged into the previous text content block. At this time, the text content block can be allowed to exceed the maximum character limit. The processing logic for tables is the same as that for images.

[0095] In addition, the second text content generated by the image can also be used as a separate text content block. For these text content blocks composed of image-generated text content, there is no maximum character limit. Based on this, it can be ensured that the second text content generated by the same image is always in the same text content block, thereby ensuring the integrity of the image-generated text content.

[0096] Based on this, the first text content and the second text content are divided into blocks according to different rules, which not only ensures the block integrity of the image-generated text content, but also takes into account the preservation of the contextual connection between the image and the text, thereby effectively improving the accuracy and flexibility of multi-model knowledge retrieval.

[0097] In some possible embodiments, step S2, parsing the image data and acquiring the second text content corresponding to the image data, may include at least one of the following:

[0098] S201, recognizing text content in the image data based on optical character recognition technology to obtain second text content corresponding to the image data;

[0099] S202: Process the image data based on the pre-trained visual language model to obtain second text content corresponding to the image data output by the visual language model.

[0100] Specifically, when the image data contains a lot of text information, optical character recognition technology can be directly used to perform text recognition on it as the second text content corresponding to the image data.

[0101] When the image data contains less text or no text information, a pre-trained visual language model can be used to understand the image data and generate corresponding second text content.

[0102] It can be understood that in some embodiments, the two methods can be combined to identify and process the same image data. For example: first, the text content in the image data is identified based on optical character recognition technology to obtain initial text content corresponding to the image data; then, the image data is processed based on a pre-trained visual language model to obtain auxiliary text content corresponding to the image data output by the visual language model; and the second text content corresponding to the image data is obtained by splicing the initial text content and the auxiliary text content.

[0103] In some possible embodiments, step S1, extracting first text content and image data of the original document, may include:

[0104] S101, extracting first text content and image data of an original document, and recording first position information of the first text content and second position information of the image data;

[0105] Step S3, combining the first text content and the second text content to obtain mixed text content corresponding to the original document, may include:

[0106] S301. Sort and combine first text content and second text content based on first position information and second position information to obtain mixed text content corresponding to an original document.

[0107] It should be noted that, when extracting the first text content and image data of the original document, the position information of each part of the data content relative to the original document may also be recorded.

[0108] Exemplarily, the location information of the first text content may include the page number, the paragraph number, or specific coordinate information (eg, the coordinates corresponding to the first word in the paragraph), etc.

[0109] Exemplarily, the location information of the image data may include the page number where the image is located, specific coordinate information (such as the center point or the upper left corner point), etc. It is understandable that, since one image corresponds to one second text content, the location information of each second text content is equal to the location information of the corresponding image data.

[0110] Then, in the process of combining the first text content and the second text content, the elements (first text content or image data) are sorted and combined in order according to the position information corresponding to each element (first text content or image data) to obtain the mixed text content corresponding to the original document. Among them, for elements belonging to the same page number, they can be sorted from top to bottom and from left to right according to the specific coordinate information according to actual needs, so that the combined mixed text content conforms to the content layout structure of the original document.

[0111] Please refer to Figure 3 , Figure 3 FIG. 1 is a block diagram showing a composition of a multimodal knowledge base management device provided in some embodiments of the present application. It should be understood that the multimodal knowledge base management device is similar to the above-mentioned Figure 1 Corresponding to the method embodiment, the various steps involved in the above method embodiment can be executed. The specific functions of the management device of the multimodal knowledge base can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here.

[0112] Figure 3 The management device of the multimodal knowledge base includes at least one software function module that can be stored in a memory or fixed in the management device of the multimodal knowledge base in the form of software or firmware, and the management device of the multimodal knowledge base includes:

[0113] A data extraction module 310, for extracting first text content and image data of an original document;

[0114] An image parsing module 320, configured to parse the image data and obtain second text content corresponding to the image data;

[0115] A text combining module 330, configured to combine the first text content and the second text content to obtain mixed text content corresponding to the original document;

[0116] The text segmentation module 340 is used to segment the mixed text content into blocks to obtain a corresponding text content block set;

[0117] The knowledge base construction module 350 is used to construct a multimodal knowledge base based on a set of text content blocks.

[0118] It can be understood that the above-mentioned device item embodiment corresponds to the method item embodiment of the present invention. A multimodal knowledge base management device provided by the embodiment of the present invention can implement the multimodal knowledge base management method provided by any method item embodiment of the present invention.

[0119] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method, and will not be described in detail here.

[0120] like Figure 4 As shown, some embodiments of the present application provide an electronic device 400, which includes: a memory 410, a processor 420, and a computer program stored in the memory 410 and executable on the processor 420, wherein the processor 420 reads the program from the memory 410 through a bus 430 and executes the program to implement the method of any embodiment of the management method of the multimodal knowledge base as described above.

[0121] Processor 420 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 420 can be a microprocessor.

[0122] The memory 410 may be used to store instructions executed by the processor 420 or data related to the execution of instructions. These instructions and / or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 420 of the disclosed embodiment may be used to execute instructions in the memory 410 to implement the method shown above. The memory 410 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

[0123] Some embodiments of the present application further provide a computer-readable storage medium having a computer program stored thereon. The computer program is executed by a processor to execute the method described in the method embodiment.

[0124] Some embodiments of the present application further provide a computer program product, which, when executed on a computer, enables the computer to execute the method described in the method embodiment.

[0125] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0126] In several embodiments provided by the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0128] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0129] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0130] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0131] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A method for managing a multimodal knowledge base, characterized in that: include: extracting first text content and image data of the original document; parsing the image data and acquiring second text content corresponding to the image data; Combining the first text content and the second text content to obtain mixed text content corresponding to the original document; Processing the mixed text content in blocks to obtain a corresponding text content block set; A multimodal knowledge base is constructed based on the text content block set.

2. The method for managing a multimodal knowledge base according to claim 1, characterized in that: After constructing the multimodal knowledge base based on the text content block set, the method further includes: In response to a question query request, at least one target text content block matching the question query request is selected from the text content block set; An answer result corresponding to the question query request is determined based on the at least one target text content block.

3. The method for managing a multimodal knowledge base according to claim 2, characterized in that: Each text content block in the text content block set is stored in the form of a text vector in a vector database corresponding to the multimodal knowledge base; The step of responding to the question query request and selecting at least one target text content block matching the question query request from the text content block set includes: Responding to a question query request, and obtaining a query text vector corresponding to the question query request; Traversing and calculating the vector similarity between the query text vector and the text vectors of each of the text content blocks; Based on the vector similarities corresponding to the text content blocks, a text content block that meets a preset vector similarity condition is used as at least one target text content block that matches the question query request.

4. The method for managing a multimodal knowledge base according to claim 2, characterized in that: Also includes: In the case where it is determined that the at least one target text content block contains the second text content, obtaining target image data corresponding to the at least one target text content block based on a pre-stored image-text mapping relationship; wherein the image-text mapping relationship is a mapping relationship between pre-recorded image data and its corresponding second text content; The target image data is output as a reference answer result corresponding to the question query request.

5. The method for managing a multimodal knowledge base according to claim 1, characterized in that: The mixed text content is processed into blocks to obtain a corresponding set of text content blocks, including: The first text content in the mixed text content is processed in blocks according to a preset upper limit on the number of characters, and the second text content in the mixed text content is processed in blocks according to unit image data as a whole block to obtain a corresponding text content block set.

6. The method for managing a multimodal knowledge base according to claim 1, characterized in that: The parsing the image data and acquiring the second text content corresponding to the image data includes: Recognize the text content in the image data based on optical character recognition technology to obtain second text content corresponding to the image data; Alternatively, the image data is processed based on a pre-trained visual language model to obtain second text content corresponding to the image data output by the visual language model.

7. The method for managing a multimodal knowledge base according to claim 1, characterized in that: The step of extracting the first text content and image data of the original document comprises: Extracting first text content and image data of the original document, and recording first position information of the first text content and second position information of the image data; The combining the first text content and the second text content to obtain mixed text content corresponding to the original document includes: The first text content and the second text content are sorted and combined based on the first position information and the second position information to obtain mixed text content corresponding to the original document.

8. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can implement the multimodal knowledge base management method described in any one of claims 1 to 7 when executing the program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for managing a multimodal knowledge base according to any one of claims 1 to 7 is executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method for managing a multimodal knowledge base according to any one of claims 1 to 7 is implemented.

Images