Multi-modal knowledge base construction and retrieval method and system, storage medium and program product

Abstract

The invention provides a multi-modal knowledge base construction and retrieval method and system, a storage medium and a program product, and relates to the technical field of computers. The multi-modal knowledge base construction method comprises the following steps of obtaining original data, and performing structured knowledge splitting on the original data to obtain structured knowledge units; performing knowledge recombination on the structured knowledge unit to obtain a multi-level index; and performing multi-modal knowledge integration on the structured knowledge units according to the multi-level indexes to obtain a multi-modal knowledge base. According to the method, unstructured data can be converted into multi-modal structured data, and algorithm module functions and related knowledge such as multi-modal knowledge base construction and multi-modal knowledge retrieval can be achieved. The whole multi-modal knowledge base construction and retrieval process is high in credibility and expansibility, and the overall efficiency of knowledge management can be obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multimodal knowledge base construction and retrieval method, system, storage medium and program product. Background Art

[0002] In multiple industries, various companies and researchers generally encounter a series of common problems in knowledge management. These problems become more and more prominent as the complexity of the work increases. In the process of processing various types of materials such as technical documents and audio and conducting knowledge management, they encounter problems such as systematic knowledge loss and multimodal knowledge omissions.

[0003] The lack of systematic knowledge is mainly reflected in the fact that the knowledge contained in the data is usually in an unstructured form, which leads to inefficiency in the knowledge management and retrieval process. Due to the lack of necessary organization and association, this information cannot form a complete system, making it difficult for managers to quickly and accurately obtain the required information when facing complex challenges.

[0004] The problem of multimodal knowledge omissions stems from the fact that the knowledge modalities of data are diverse, not limited to pure text descriptions, but also include a large number of flowcharts, data tables, audio files, video clips, etc. These diversified information carriers often carry high-value content, but in the traditional knowledge management system, they have not been effectively integrated and utilized.

[0005] In response to the above problems, a new type of knowledge base is urgently needed to solve problems such as systematic knowledge loss and multimodal knowledge omissions, in order to enhance the accessibility and practicality of knowledge and optimize the overall effect of knowledge management. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a knowledge unit splitting, reorganization, multimodal knowledge integration and retrieval method based on multimodal model technology, aiming to transform unstructured data into multimodal structured data, multimodal knowledge base construction, multimodal knowledge retrieval and other algorithm module functions and related knowledge. The entire multimodal knowledge base construction and retrieval process is highly credible and extensible, which can significantly improve the overall efficiency of knowledge management.

[0007] In a first aspect, the present invention provides a method for constructing a multimodal knowledge base, comprising the following steps:

[0008] Obtain original data, perform structured knowledge decomposition on the original data, and obtain structured knowledge units;

[0009] Reorganize structured knowledge units to obtain multi-level indexes;

[0010] According to the multi-level index, multimodal knowledge integration is performed on the structured knowledge units to obtain a multimodal knowledge base.

[0011] As a further improvement of the present invention, the original data includes documents, charts, and audio.

[0012] As a further improvement of the present invention, the method for splitting the structured knowledge of the document includes:

[0013] The original materials include documents and audio;

[0014] Paginate the document and configure the prompt word template;

[0015] Use a multimodal large model to analyze the document, read and split the directory outline, text elements, image elements and table elements in the document;

[0016] Extract keywords and summarize text elements;

[0017] The directory outline, text elements, image elements and table elements are structured and organized to obtain structured document knowledge units.

[0018] As a further improvement of the present invention, the method for splitting the structured knowledge of the audio includes:

[0019] Use automatic speech recognition models to convert audio to audio text;

[0020] Analyze audio text, extract and generate audio outline, audio summary and audio keywords;

[0021] The audio outline, audio summary and audio keywords are structured to obtain structured audio knowledge units.

[0022] As a further improvement of the present invention, the structured knowledge units are reorganized to obtain a multi-level index, including:

[0023] Analyze the logical connections between structured knowledge units;

[0024] Based on the analysis results of logical associations, a tree-like multi-level index structure is constructed using a full-text search engine.

[0025] As a further improvement of the present invention, the logical association includes causal relationship, time sequence and subject similarity.

[0026] As a further improvement of the present invention, the multi-level index structure includes:

[0027] Index each structured knowledge unit as a document and add corresponding metadata to it, including chapter numbers and keywords;

[0028] Add additional metadata to the audio content, including the start time, end time, and speaker information of the audio.

[0029] As a further improvement of the present invention, the multimodal knowledge integration is performed on the structured knowledge units according to the multi-level index to obtain a multimodal knowledge base, including:

[0030] Use a multimodal large model to generate deep summaries of the diagram content of structured knowledge units;

[0031] Transform the graph contents of structured knowledge units and the generated deep summaries into high-dimensional vectors to obtain multimodal vectors;

[0032] The multimodal vectors are stored in a vector database to obtain a multimodal knowledge base.

[0033] In a second aspect, the present invention provides a multimodal knowledge base retrieval method, using the multimodal knowledge base constructed by the method described in the first aspect, comprising the following steps:

[0034] The user submits a query request to the system;

[0035] The system first parses the query request and identifies the keywords in the query request;

[0036] Use keywords to perform full-text search and obtain full-text search results;

[0037] Convert the query request into a high-dimensional vector, perform vector retrieval, and obtain vector retrieval results;

[0038] The full-text search results and vector search results are merged and sorted to generate the final search results.

[0039] As a further improvement of the present invention, the final search results include: text materials, charts, flow charts, data tables, and audio files.

[0040] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect and / or the second aspect.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect and / or the second aspect.

[0042] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect and / or the second aspect.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention provides a multimodal knowledge base construction and retrieval method, system, storage medium and program product, which can transform unstructured data into multimodal structured data, multimodal knowledge base construction, multimodal knowledge retrieval and other algorithm module functions and related knowledge. The entire multimodal knowledge base construction and retrieval process is highly credible and extensible, and can significantly improve the overall efficiency of knowledge management. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flow chart of a multimodal knowledge base construction and retrieval method provided by the present invention;

[0046] Figure 2 A flow chart of a method for constructing a multimodal knowledge base provided by the present invention;

[0047] Figure 3 A flow chart of a multimodal knowledge base retrieval method provided by the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the present invention will combine the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, wherein steps S1, S2... in the embodiments described in the present invention do not limit the only execution steps of the present invention; the various models described in the present invention are not the only limiting methods of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] In the present invention, computer devices / equipment / systems refer to related entities applied to computers, such as hardware, a combination of hardware and software, software or software in execution, etc. In detail, for example, software includes but is not limited to a process running on a processor, a processor, an object, executable software, an execution thread, a program and / or a computer. In addition, an application or script program running on a server, a server can also be software. One or more software can be in an execution process and / or thread, and the software can be localized on one computer and / or distributed between two or more computers, and can be run by various computer-readable media.

[0050] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0051] The implementation methods of each module are described in detail below. Taking the construction and retrieval of a multimodal knowledge base in the field of biosafety as an example, the entire knowledge management process is explained, and the entire process can be applied to other fields.

[0052] The technical architecture of multimodal knowledge base construction consists of multiple modules, including structured knowledge unit splitting, knowledge reorganization, multimodal content understanding, multimodal knowledge integration, etc. The modules work together to form a complete multimodal knowledge unit splitting and structured knowledge reorganization process.

[0053] Similarly, the technical architecture of multimodal knowledge base retrieval also consists of multiple modules, including query parsing, hybrid retrieval integrating keyword retrieval and vector retrieval, recall result fusion, and multimodal result presentation, forming a complete multimodal knowledge unit retrieval and recall process.

[0054] In a first aspect, the present invention provides a method for constructing a multimodal knowledge base, such as Figure 1 As shown, the following steps are included:

[0055] S100: Acquire original data, perform structured knowledge decomposition on the original data, and obtain structured knowledge units;

[0056] Preferably, the original data includes documents, charts, and audio.

[0057] In a specific embodiment of the present invention, a method based on a combination of multimodal large model analysis and semantic understanding is used to effectively decompose complex technical documents into meaningful structured knowledge units. Hierarchical analysis divides documents into different logical parts based on the logical hierarchical relationships of the document, such as chapter titles and paragraph titles; through semantic understanding, it goes deep into the text and uses natural language processing technology to identify the core meaning and key words of sentences and paragraphs, thereby extracting valuable information fragments. The combination of these two methods not only retains the original logical framework of the document, but also enhances the retrievability and interconnectivity of information, providing a solid foundation for subsequent knowledge reorganization and knowledge retrieval.

[0058] Preferably, the method for splitting the structured knowledge of the document includes:

[0059] S111: The original data includes documents and audio;

[0060] S112: paging the document and configuring a prompt word template;

[0061] S113: Analyze the document using a multimodal large model, read and split the directory outline, text elements, image elements and table elements in the document;

[0062] S114: extracting keywords and summarizing text elements;

[0063] S115: Structurally organize the catalog outline, text elements, image elements, and table elements to obtain a structured document knowledge unit.

[0064] In a specific implementation of the present invention, the document is paginated, a prompt word template is configured, and the Qwen-vl-72B multimodal large model is used to analyze the document, read and split the directory, paragraphs, and chart elements in the document, and the document content is structured. At the same time, considering efficiency, the Qwen-7B model is used to extract keywords and summarize the text content, refine the information fragments in the document, and construct a rich and comprehensive document knowledge unit.

[0065] In the process of document structuring, Qwen-vl-72B, as an advanced multimodal large model, demonstrated its outstanding capabilities. It can not only efficiently identify and parse the text content in the document, but also understand and analyze non-text elements such as images and charts, which is especially important for complex documents containing a large amount of graphic materials. By configuring specific prompt word templates, Qwen-vl-72B can accurately read and split the catalog, paragraphs, and chart elements in the document to achieve highly structured organization of the document content. This process greatly improves the accessibility and usability of document information, and provides a solid foundation for subsequent knowledge management. In addition, Qwen-vl-72B also performs well in processing cross-language documents, and can support document parsing in multiple languages, meeting the diverse document processing needs in the context of globalization.

[0066] The Qwen-7B model has become an indispensable part with its efficient text processing capabilities. As a lightweight but powerful text processing model, Qwen-7B focuses on quickly extracting key information from large amounts of text data, including but not limited to keyword extraction and summary generation. This enables the core points to be captured quickly when faced with large and complex document materials. By working in collaboration with Qwen-vl-72B, Qwen-7B further refines the information fragments in the document and builds a knowledge system that is both rich in content and comprehensive. This combination not only improves the efficiency of document processing, but also ensures the accuracy and completeness of the information, and is suitable for knowledge management in multiple fields. The Qwen-7B model also has good scalability and adaptability, and parameter settings can be adjusted according to different application scenarios to achieve the best processing effect.

[0067] like Figure 2 As shown, by combining the multimodal large model with the large language model sub-module, the directory of the original document, the chart elements in the page, etc. are split into independent knowledge units, and additional information is added to the knowledge units.

[0068] Preferably, the method for splitting the structured knowledge of the audio includes:

[0069] S120: Use the automatic speech recognition model to convert the audio into audio text;

[0070] S121: Analyze the audio text, extract and generate audio outline, audio summary and audio keywords;

[0071] S122: Structurally organize the audio outline, audio summary and audio keywords to obtain a structured audio knowledge unit.

[0072] In a specific embodiment of the present invention, in order to process audio content more accurately, the present invention introduces the open source automatic speech recognition (ASR) model DeepSpeech. DeepSpeech is a high-performance, open source speech recognition engine developed by Mozilla. It is based on deep learning technology and can efficiently and accurately convert audio into text. DeepSpeech supports multiple languages, has strong robustness and flexibility, and can maintain high recognition accuracy in various environments. This process is not limited to simple transcription, but can also identify subtle features such as the identity of the speaker, providing a richer information basis for subsequent semantic understanding.

[0073] Through the application of the Qwen-7B model, the converted text is deeply understood and analyzed. Qwen-7B can not only identify the key information and core concepts of dialogues and narrations, but also generate detailed audio outlines. These outlines divide the audio content into multiple structured parts according to chronological order and content logic, and each part contains main points and important details. In addition, Qwen-7B will also extract keywords and generate summaries to further refine information and construct content-rich and comprehensive knowledge units. In this way, similar to the structured knowledge splitting of documents and charts, the present invention realizes the structured knowledge splitting of unstructured audio, such as Figure 2 As shown in the figure, the DeepSpeech model works together with the Qwen-7B model to convert raw audio into knowledge units containing rich semantic information.

[0074] S200: Reorganize the structured knowledge units to obtain a multi-level index;

[0075] In a specific embodiment of the present invention, the function of the knowledge reorganization module is to reorganize the structured knowledge units into a tree-like hierarchical index. The module receives the structured knowledge units processed in the previous stage as input. By analyzing the logical associations between the units, the module constructs a multi-level index structure, which not only reflects the hierarchical relationship between the knowledge units, but also facilitates the rapid location of information. The reconstructed index system can effectively improve the efficiency and accuracy of information retrieval, while ensuring the coherence and consistency of the content between the knowledge units, thereby enhancing the quality and effect of knowledge query and content generation based on the index system.

[0076] Preferably, the structured knowledge units are reorganized to obtain a multi-level index, including:

[0077] S210: Analyze the logical connections between structured knowledge units;

[0078] Preferably, the logical association includes causal relationship, time sequence and subject similarity.

[0079] In a specific embodiment of the present invention, the structured knowledge units processed in the structured knowledge splitting stage are first received. These units have been preliminarily parsed and structured through a multimodal large model (such as Qwen-VL) and a natural language processing model (such as Qwen-7B), and contain content in various forms such as text, graphics, and audio. The Qwen-7B model is used to perform logical association analysis between knowledge units. Qwen-7B not only has efficient information extraction capabilities, but also has powerful semantic understanding and reasoning capabilities. By inputting the contents of multiple knowledge units, Qwen-7B can identify the logical relationships between them, such as causal relationships, time sequence, topic similarity, etc.

[0080] S220: Based on the analysis results of the logical association, a tree-like multi-level index structure is constructed using a full-text search engine.

[0081] Preferably, the multi-level index structure includes:

[0082] S221: Index each structured knowledge unit as a document and add corresponding metadata to it, including chapter number and keywords;

[0083] S222: Add additional metadata to the audio content, including the start time, end time, and speaker information of the audio.

[0084] In a specific embodiment of the present invention, based on the results of logical association analysis, a multi-level index structure is constructed using the open source full-text search engine Elasticsearch. Elasticsearch is a distributed search and analysis engine that can efficiently process large-scale data sets and support complex full-text search and real-time data analysis.

[0085] In Elasticsearch, the present invention indexes each knowledge unit as a document and adds corresponding metadata, such as chapter number, keywords, etc. For audio content, the present invention adds additional metadata, such as the start time, end time, speaker information, etc. of the audio.

[0086] S300: According to the multi-level index, multimodal knowledge integration is performed on the structured knowledge units to obtain a multimodal knowledge base.

[0087] Preferably, the multimodal knowledge integration is performed on the structured knowledge units according to the multi-level index to obtain a multimodal knowledge base, including:

[0088] S310: Generate deep summaries of the diagram contents of structured knowledge units using a multimodal large model;

[0089] In a specific embodiment of the present invention, a multimodal large model is used to generate in-depth summaries of the content of the chart. The present invention chooses to use the Qwen-VL-7B model because it has higher efficiency than the Qwen-VL-72B model in the summary task and is more suitable for quickly generating high-quality chart summaries. Qwen-VL-7B will perform content recognition on the input chart, including text extraction in the image, chart type recognition (such as bar charts, line charts, etc.) and position and relationship analysis of each element in the chart. After the content of the chart is accurately identified, Qwen-VL-7B will combine this information with the logical association of the knowledge units previously constructed by the Qwen-7B model to understand the interaction between the chart information and the surrounding text content. Based on a comprehensive understanding of the content of the chart and its context, corresponding prompt words are constructed, and concise and accurate chart summaries are generated through Qwen-VL-7B. These summaries not only summarize the intuitive content of the chart, but also point out the key connections between the chart content and other knowledge units.

[0090] S320: converting the graph content of the structured knowledge unit and the generated deep summary into a high-dimensional vector to obtain a multimodal vector;

[0091] S330: Storing the multimodal vector into a vector database to obtain a multimodal knowledge base.

[0092] In a specific embodiment of the present invention, after constructing knowledge units with associated logic, in order to further improve the availability and intelligence level of these knowledge units, the present invention uses the CLIP model for multimodal vector embedding. CLIP (Contrastive Language–Image Pre-training) is a multimodal pre-training model proposed by OpenAI. It uses a contrastive learning method to simultaneously train image and text encoders so that it can map data of different modalities into the same high-dimensional vector space. The advantage of this model is that it can not only effectively capture the visual information in the image, but also understand the text information related to it, thereby achieving cross-modal understanding and matching. The CLIP model is used to convert charts and their corresponding descriptive summaries into high-dimensional vectors. These vectors not only contain the visual features of the charts, but also incorporate the semantic information of the text description. In this way, knowledge content of different modalities is mapped to the same vector space, so that they can be compared and associated under the same framework.

[0093] The generated multimodal vectors are stored in the Milvus vector database. Milvus is an open source vector similarity search engine designed for efficient storage and retrieval of large-scale vector data. This vector storage mode supports a variety of distance metrics, such as Euclidean distance, cosine similarity, etc., and can quickly query the most similar records of vectors. This mode can efficiently manage and retrieve large amounts of multimodal vector data.

[0094] In a specific embodiment of the present invention, after constructing a knowledge base including Elasticsearch index and Milvus vector library of CLIP vector, the present invention efficiently returns relevant multimodal knowledge units according to the user's query request through hybrid multimodal retrieval technology. This process fully utilizes the full-text search capability of Elasticsearch and the vector similarity search capability of Milvus to achieve cross-modal information matching and retrieval.

[0095] In a second aspect, the present invention provides a multimodal knowledge base retrieval method, such as Figure 1 As shown, the multimodal knowledge base constructed using the method described in the first aspect includes the following steps:

[0096] S400: The user submits a query request to the system;

[0097] S500: The system first parses the query request and identifies keywords in the query request;

[0098] In a specific embodiment of the present invention, when a user submits a query request, the system first parses the query and identifies keywords in the query, etc. This step not only involves traditional text processing technology, but may also include preliminary processing of non-text information such as images and voices contained in the query.

[0099] For the text information in the query request, the present invention uses the built-in word segmenter and analyzer of Elasticsearch to extract keywords. The word segmenter of Elasticsearch can split the query string into separate words, and the analyzer can further process these words, such as removing stop words and performing stem extraction.

[0100] For non-text information such as pictures and voice, the present invention uses corresponding multimodal models for processing. For example, for image queries, the system can use the CLIP model to extract the visual features of the image; for voice queries, the system can use a speech recognition model (such as DeepSpeech) to convert the voice into text, and then perform keyword extraction.

[0101] like Figure 3 As shown, the query request is voice information. The voice is first converted into text, and then keywords are extracted through the relevant components of Elasticsearch to obtain keywords such as "biohazard sign" and "drawing".

[0102] S600: Perform a full-text search using the keyword to obtain a full-text search result;

[0103] In a specific embodiment of the present invention, the parsed query keywords are sent to Elasticsearch for full-text search. Elasticsearch is an efficient distributed search and analysis engine that can quickly process large-scale data sets and support complex full-text search and real-time data analysis. In Elasticsearch, each knowledge unit is indexed as a document and comes with rich metadata, such as Figure 3 As mentioned above, for audio content, the present invention also adds additional metadata, such as the start time, end time and speaker information of the audio, etc. These metadata help Elasticsearch locate and retrieve relevant information more accurately.

[0104] S700: Convert the query request into a high-dimensional vector, perform vector search, and obtain a vector search result;

[0105] In a specific embodiment of the present invention, the query request is converted into a high-dimensional vector through the CLIP model. The CLIP model not only captures the visual features of the image, but also integrates the semantic information of the text description. Based on the above, Milvus is an efficient storage and retrieval system designed specifically for large-scale vector data, supporting multiple distance measurement methods and being able to quickly find the records most similar to the query vector.

[0106] S800: Merge and sort the full-text search results and the vector search results to generate final search results.

[0107] Preferably, the final search results include: text materials, charts, flow charts, data tables, and audio files.

[0108] In a specific embodiment of the present invention, the results retrieved from Elasticsearch and Milvus are merged and sorted to generate the final search results. Figure 3 As shown, the system will assign weights to knowledge units from different sources based on the user's query intent and contextual information to ensure that the returned information is both comprehensive and accurate.

[0109] The final search results of the present invention include not only text materials, such as Figure 3 As shown, it can also include multimodal resources such as charts, flowcharts, data tables, audio files, etc. These results will be presented in a user-friendly way, including rich and comprehensive knowledge unit information, helping users quickly locate the required knowledge sources and other auxiliary information.

[0110] In a third aspect, the present invention provides a computer system comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect and / or the second aspect.

[0111] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect and / or the second aspect.

[0112] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect and / or the second aspect.

Claims

1. A method for constructing a multimodal knowledge base, characterized in that: The following steps are involved: Obtain original data, perform structured knowledge decomposition on the original data, and obtain structured knowledge units; Reorganize structured knowledge units to obtain multi-level indexes; According to the multi-level index, multimodal knowledge integration is performed on the structured knowledge units to obtain a multimodal knowledge base.

2. The method for constructing a multimodal knowledge base according to claim 1, characterized in that: The original data include documents, charts, and audio.

3. The method for constructing a multimodal knowledge base according to claim 2, characterized in that: The method for splitting the structured knowledge of the document includes: The original materials include documents and audio; Paginate the document and configure the prompt word template; Use a multimodal large model to analyze the document, read and split the directory outline, text elements, image elements and table elements in the document; Extract keywords and summarize text elements; The directory outline, text elements, image elements and table elements are structured and organized to obtain structured document knowledge units.

4. The method for constructing a multimodal knowledge base according to claim 2, characterized in that: The structured knowledge splitting method for the audio includes: Use automatic speech recognition models to convert audio to audio text; Analyze audio text, extract and generate audio outline, audio summary and audio keywords; The audio outline, audio summary and audio keywords are structured to obtain structured audio knowledge units.

5. The method for constructing a multimodal knowledge base according to claim 1, characterized in that: The structured knowledge units are reorganized to obtain a multi-level index, including: Analyze the logical connections between structured knowledge units; Based on the analysis results of logical associations, a tree-like multi-level index structure is constructed using a full-text search engine.

6. The method for constructing a multimodal knowledge base according to claim 5, characterized in that: The logical connections include causality, temporal order, and subject similarity.

7. The method for constructing a multimodal knowledge base according to claim 5, characterized in that: The multi-level index structure includes: Index each structured knowledge unit as a document and add corresponding metadata to it, including chapter numbers and keywords; Add additional metadata to the audio content, including the start time, end time, and speaker information of the audio.

8. The method for constructing a multimodal knowledge base according to claim 1, characterized in that: According to the multi-level index, multi-modal knowledge integration is performed on the structured knowledge units to obtain a multi-modal knowledge base, including: Use a multimodal large model to generate deep summaries of the diagram content of structured knowledge units; Transform the graph contents of structured knowledge units and the generated deep summaries into high-dimensional vectors to obtain multimodal vectors; The multimodal vectors are stored in a vector database to obtain a multimodal knowledge base.

9. A multimodal knowledge base retrieval method, characterized in that: The multimodal knowledge base constructed using the method according to any one of claims 1 to 8 comprises the following steps: The user submits a query request to the system; The system first parses the query request and identifies the keywords in the query request; Use keywords to perform full-text search and obtain full-text search results; Convert the query request into a high-dimensional vector, perform vector retrieval, and obtain vector retrieval results; The full-text search results and vector search results are merged and sorted to generate the final search results.

10. The multimodal knowledge base retrieval method according to claim 9, characterized in that: The final search results include: text materials, charts, flow charts, data tables, and audio files.

11. A computer system comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1-10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images