Multimodal database construction method and device based on visual language model collaborative routing

Abstract

The embodiment of the specification provides a multi-modal database construction method and device based on a visual language model collaborative routing, wherein the method comprises the following steps: analyzing original document data to obtain multi-modal element data, and determining element structure information corresponding to the multi-modal element data; arranging the multi-modal element data based on the element structure information to obtain target document data; splitting the target document data to obtain a structured data set, and determining target key data according to structured data in the structured data set; and constructing a multi-modal database according to the target key data, wherein the multi-modal database is used for data retrieval. The embodiment realizes efficient processing of multi-modal unstructured data, improves the database construction efficiency, and at the same time enables subsequent data retrieval based on the multi-modal database to reduce the retrieval time and improve the retrieval efficiency.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the field of information technology, in particular to a multi-modal database construction method and device based on visual language model collaborative routing. BACKGROUND

[0002] With the development of artificial intelligence technology, large language models are pre-trained through massive texts and exhibit excellent generation capabilities in tasks such as question answering, summarization, and translation. However, traditional generative models have three key bottlenecks in the problem generation scenario: knowledge limitation (relying on static training data, unable to dynamically acquire new knowledge), model illusion (generating seemingly reasonable but actually incorrect content), and insufficient explainability (lack of traceable generation basis). Therefore, retrieval-augmented generation (RAG) technology has emerged, which aims to improve the accuracy of information retrieval and the credibility of generated content by combining external knowledge bases and generation models to overcome the limitations of large language models (LLM) in text generation tasks. However, existing RAG technology still faces challenges in processing multi-modal data, dynamic updating capabilities, and retrieval efficiency. Therefore, how to improve the efficiency of multi-modal data processing and provide an efficient retrieval method is a problem that needs to be solved. SUMMARY

[0003] Therefore, the embodiments of the present specification provide a multi-modal database construction method based on visual language model collaborative routing. One or more embodiments of the present specification also relate to a multi-modal database construction device based on visual language model collaborative routing, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects in the prior art.

[0004] According to a first aspect of the embodiments of the present specification, a multi-modal database construction method based on visual language model collaborative routing is provided, comprising:

[0005] parsing the original document data to obtain multi-modal element data, and determining element structure information corresponding to the multi-modal element data;

[0006] arranging the multi-modal element data based on the element structure information to obtain target document data;

[0007] segmenting the target document data to obtain a structured data set, and determining target key data according to structured data in the structured data set;

[0008] constructing a multi-modal database according to the target key data, wherein the multi-modal database is used for data retrieval.

[0009] According to a second aspect of the embodiments of the present specification, a multi-modal database construction device based on visual language model collaborative routing is provided, comprising:

[0010] The parsing module is configured to parse the original document data to obtain multi-modal element data, and determine element structure information corresponding to the multi-modal element data;

[0011] The arrangement module is configured to arrange the multi-modal element data based on the element structure information, and obtain target document data;

[0012] The cutting module is configured to cut the target document data to obtain a structured data set, and determine target key data according to structured data in the structured data set;

[0013] The adding module is configured to add the target key data to a multi-modal database, wherein the multi-modal database is used for data retrieval.

[0014] According to a third aspect of the embodiments of the present specification, a computing device is provided, comprising:

[0015] a memory and a processor;

[0016] The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which implement the steps of the above-mentioned multi-modal database construction method based on visual language model collaborative routing when executed by the processor.

[0017] According to a fourth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, which implement the steps of the above-mentioned multi-modal database construction method based on visual language model collaborative routing when executed by the processor.

[0018] According to a fifth aspect of the embodiments of the present specification, a computer program product is provided, comprising a computer program or instructions, which implement the steps of the above-mentioned multi-modal database construction method based on visual language model collaborative routing when executed by the processor.

[0019] One embodiment of the present specification realizes parsing of original document data to obtain multi-modal element data, so that subsequent processing of the original document data can be more comprehensive. The element structure information of the multi-modal element data is determined, and the multi-modal element data is arranged based on the element structure information, so that the obtained target document data can retain the structure information of the original document data, facilitating subsequent segmentation of the target document data. The structured data set is obtained by segmenting the target document data, which can obtain more detailed structured data and ensure data integrity, avoiding data fragmentation problems. The target key data is determined according to the structured data in the structured data set, and the multi-modal database is constructed according to the target key data, realizing efficient processing of multi-modal unstructured data, improving the database construction efficiency, and at the same time, reducing the search time and improving the search efficiency when searching data based on the multi-modal database subsequently. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 is a flowchart of a multi-modal database construction method based on a visual language model collaborative routing provided by one embodiment of the present specification;

[0021] Figure 2 is a system framework schematic diagram of a multi-modal database construction method based on a visual language model collaborative routing provided by one embodiment of the present specification;

[0022] Figure 3 is a processing process flowchart of a multi-modal database construction method based on a visual language model collaborative routing provided by one embodiment of the present specification;

[0023] Figure 4 is a structural schematic diagram of a multi-modal database construction device provided by one embodiment of the present specification;

[0024] Figure 5 is a structural block diagram of a computing device provided by one embodiment of the present specification. DETAILED DESCRIPTION

[0025] In the following description, many specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced in many different ways beyond the specific embodiments described herein, and it is understood that persons having ordinary skill in the art can make similar modifications to the present specification without departing from the spirit of the present specification, so the present specification is not limited to the specific implementations disclosed below.

[0026] The terminology used in this description of one or more embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments described in this description. As used in this description of one or more embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0027] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, a first item could be termed a second item, and, similarly, a second item could be termed a first item without departing from the scope of one or more embodiments described herein. As used herein, the term “if’ can be construed to mean “when” or “if” in some embodiments. That is, if a condition is met, then a given event will occur, or, if a condition is not met, then a given event will not occur.

[0028] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present description are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0029] First, the terms involved in one or more embodiments of the present description are explained.

[0030] LLM: Large Language Model (LLM) is an artificial intelligence model based on deep learning technology, trained on a large amount of corpus, with strong language understanding and generation capabilities. LLM can understand and generate natural language, complete tasks such as question answering, writing, translation, reasoning, programming, etc.

[0031] VLM: Vision-Language Model (VLM) is a multi-modal artificial intelligence model that can understand and process visual information (images, videos) and natural language (text) simultaneously, and establish semantic association between the two. The goal of such models is to achieve cross-modal understanding and generation capabilities.

[0032] OCR: Optical Character Recognition (OCR) is a technology that can convert text in various types of documents, such as scanned paper documents, image portions in PDF files, etc., into computer-editable text. OCR technology greatly improves information processing efficiency, enabling the digitization of text content in paper documents or images for further processing and analysis.

[0033] RAG: Retrieval-Augmented Generation (RAG) is a technical framework that combines information retrieval and text generation, aiming to overcome the limitations of traditional Large Language Models (LLM) in knowledge limitations, model hallucinations, and explainability. RAG enhances the capabilities of generation models by integrating external knowledge bases, making generated content more accurate, relevant, and credible. RAG systems typically include: RAG-Index: building and maintaining an efficient knowledge base index, directly determining the quality of knowledge data and affecting the diversity and efficiency of queries in the retrieval stage, and the reliability of content in the generation stage; RAG-Retrieval: retrieving relevant information from the constructed index to support the generation task. This process uses retrieval techniques to find context or background knowledge related to the input; RAG-Generation: generating new content or answers based on the retrieved information. Here, generative models such as large-scale pre-trained language models with Transformer architecture are used.

[0034] Natural language processing, as one of the core fields of artificial intelligence, aims to give machines the ability to understand, generate, and process human language. In recent years, large language models have shown excellent generation capabilities in tasks such as question answering, summarization, and translation through massive text pre-training. However, traditional generative models have three key bottlenecks in the problem generation scenario: knowledge limitations (relying on static training data, unable to dynamically acquire new knowledge), model hallucinations (generating seemingly reasonable but actually incorrect content), and insufficient explainability (lack of traceable generation basis). Therefore, RAG technology emerges as the times require, by combining external knowledge bases with generation models, injecting real-time retrieval context information into LLM, significantly improving the accuracy, credibility, and timeliness of generated content.

[0035] With the explosive growth of multi-modal data (text, images, tables, etc.), RAG-index technology has gradually expanded from single-text scenarios to multi-modal scenarios. However, existing solutions face severe challenges in multi-modal processing, such as the following defects of RAG-index technology in multi-modal scenarios:

[0036] Low processing efficiency of multi-source heterogeneous data: To realize the parsing of heterogeneous nonlinear data into structured data, and the semantic alignment of information of different modalities (such as text description, image features, video) into a unified context representation structure. Text parsing methods such as TextRAG only focus on text information, directly ignoring the semantic information of images, tables, etc. Model-based text parsing methods such as Transformer-TextRAG realize modality agreement by reducing the dimensionality of visual elements to text vector controls, but the irreversible loss of fine-grained semantics such as image layout and entity relationship is caused by the dimensionality reduction process. Although VLM-RAG based on visual language model VLM can preserve multi-modal semantics, the high retrieval overhead brought by the full-modal cross-attention calculation makes it difficult to meet the real-time needs of industrial level.

[0037] Knowledge fragmentation contradiction: In order to improve the retrieval accuracy, the document data needs to be divided into multiple smaller data blocks; in the generation of large models, in order to refer to more information, larger data blocks are needed. This will cause the problem of knowledge fragmentation contradiction. TextRAG, Transformer-TextRAG and other technologies directly split the text based on semantic similarity, and the size of the knowledge block is difficult to control; VLM-RAG divides each page of document data as a knowledge block, which may cause data fragmentation between knowledge blocks.

[0038] The above problems indicate that the problem of RAG-index in the multi-modal scene lies in the balance between semantics and efficiency. If the semantic integrity (such as preserving fine-grained information such as image layout and table structure) is pursued, complex model calculations are needed, and the retrieval efficiency is sacrificed; if the calculation efficiency is optimized first, the generation quality may be reduced due to modal dimensionality reduction or information fragmentation.

[0039] Based on this, in the present specification, a multi-modal database construction method based on visual language model collaborative routing is provided, which aims to solve the technical problem of the above-mentioned multi-modal semantic-efficiency trade-off dilemma. The present specification also relates to a multi-modal database construction method based on visual language model collaborative routing, a computing device, a computer readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0040] Referring to Figure 1 , Figure 1 A flowchart of a multi-modal database construction method based on visual language model collaborative routing according to one embodiment of the present specification is shown, which specifically includes the following steps.

[0041] Step 102: Analyzing the original document data to obtain multi-modal element data, and determining the element structure information corresponding to the multi-modal element data.

[0042] Step 104: arranging the multi-modal element data based on the element structure information to obtain target document data.

[0043] Step 106: splitting the target document data to obtain a structured data set, and determining target key data according to the structured data in the structured data set.

[0044] Step 108: constructing a multi-modal database according to the target key data, wherein the multi-modal database is used for data retrieval.

[0045] Among them, the original document data includes various types of document data such as text, pictures, tables, etc., that is, the original document data can be understood as unstructured document data. It is necessary to intelligently analyze the original document data and extract multi-modal element data from the original document data. Multi-modal element data can be understood as various types of document data contained in the original document data, and can include text element data, picture element data, table element data, etc. After obtaining the multi-modal element data, the element structure information of the multi-modal element data needs to be determined. The element structure information can be understood as the hierarchical structure information of the multi-modal element data in the original document data. According to the element structure information, the original logical level of the multi-modal element data can be determined, so as to ensure that the hierarchical structure of the original document data is not lost. Subsequently, the multi-modal element data can be arranged based on the element structure information to obtain target document data. The target document data can be understood as document data obtained by structuring and arranging the multi-modal element data according to the determined hierarchical structure, that is, the target document data belongs to structured documents, which provides high-quality structured data support for subsequent knowledge retrieval and generation. Subsequently, the target document data is split to obtain a structured data set, so as to obtain a smaller data unit. In the splitting process, the hierarchical structure and semantic relationship of the data also need to be considered, so as to avoid the problem of data content fragmentation and facilitate subsequent data retrieval and model processing. According to the structured data in the structured data set, the target key data is determined. The target key data can be understood as key information extracted from the structured data. These key information can reduce the data amount on the premise of ensuring the structure and semantic integrity of the data, and improve the subsequent retrieval efficiency. According to the target key data, a multi-modal database can be constructed. Subsequently, data retrieval can be performed on the multi-modal database, so as to realize cross-modal retrieval of multi-modal data using the multi-modal database, and also solve the corresponding semantic-efficiency balance problem.

[0046] Referring to Figure 2 , Figure 2is a system framework diagram of a multi-modal database construction method based on a visual language model collaborative routing provided by an embodiment of the present specification, wherein the system framework diagram includes a parsing-arranging unit, a splitting-extracting unit, and a storage-construction unit. The parsing-arranging unit can parse the original document data into multi-modal element data, and arrange the multi-modal element data into target document data based on element structure information. The target document data is split into a structured data set by the splitting-extracting unit, and then target key data is extracted from the structured data in the structured data set. The multi-modal database is constructed based on the target key data by the storage-construction unit.

[0047] In practical applications, the parsing-arranging unit realizes intelligent parsing of unstructured documents through a multi-stage pipeline. First, based on layout analysis technology, text, pictures, tables, and noise elements are located and classified. Combined with OCR and visual models, multi-modal content is extracted (text boxes are converted into editable text, and pictures and tables are stored as URLs and structured into Markdown format). At the same time, a Doc-Tree hierarchical analysis algorithm based on large model classification is designed. By fusing physical characteristics such as element position, line width, and page number with semantic understanding capabilities, the title level is dynamically identified and the document tree structure is constructed. Finally, semi-structured Markdown documents with preserved original logical levels and denoised are output, providing high-quality structured data support for subsequent knowledge retrieval and generation. The splitting-extracting unit converts the parsed Markdown document into structured JSON data through a hierarchical blocking strategy. First, the document logical structure is preserved by preliminary blocking according to chapter title levels. Then, under the maximum blocking length limit (chunk_size), fine-grained splitting is performed based on natural paragraphs or semantic boundaries to avoid content fragmentation. After associating the picture / table URL with the text block through coordinate matching, noise URLs are removed, and small size context blocks are integrated based on merging rules. Finally, multi-modal structured data containing text blocks (chunk.json), pictures (image.json), and tables (table.json) are output, realizing three optimizations of hierarchical structure preservation, multi-modal association, and noise filtering. Based on the multi-modal extraction strategy, text and images are extracted using subject-predicate-object structure to ensure the structure and semantic integrity of the text information. Table extraction uses key-value structure to efficiently extract key information from tables while maintaining structure. The storage-construction unit adds the extracted key information to the database or creates a database based on the key information. The multi-modal database can solve the scalability bottleneck of traditional solutions through hierarchical design and dynamic routing mechanism. A three-layer index mechanism is constructed: the knowledge graph layer supports complex reasoning queries, the semantic retrieval layer realizes cross-modal similarity matching and fast locking of knowledge graph layer elements, the collaborative document layer dynamically manages multi-modal documents and incremental updates, and related temporal association queries.

[0048] Based on this, based on the system framework of the multi-modal database construction method based on the visual language model collaborative routing provided above, the framework is designed by decoupling each unit, solving the problems of existing RAG framework in multi-modal semantic-efficiency balance, knowledge block contradiction, single index strategy, etc. Through element segmentation of the document page, storage of different elements, and construction of the document structure tree based on large model fusion of physical features and semantic features, a multi-modal document complete information analysis scheme is realized; by combining title recognition, title line width, page number and other physical features and large model reasoning ability, title level classification is realized, ensuring that the extracted data can retain the structure information of the original document; by hierarchical division based on the title structure, the logical coherence of the data content is ensured, and the semantic and block maximum limit method is used for hierarchical blocks to ensure that the divided blocks have context consistency, realizing the preservation of the document hierarchy structure and the avoidance of semantic fragmentation problem; by the divide-and-conquer coding and divide-and-conquer extraction strategy, on-demand processing and dynamic scheduling of text, image and table are realized, improving the subsequent retrieval accuracy and efficiency; the introduction of knowledge graph layer, semantic retrieval layer and collaborative document layer in the multi-modal database realizes three-layer index, solving the problem of single index architecture. The multi-modal database construction method based on the visual language model collaborative routing provided in the specification is described in detail through each embodiment.

[0049] Further, since the original document data includes various types of document data, it is necessary to analyze and extract different types of document data respectively. Specifically, the original document data is analyzed to obtain multi-modal element data, including: content recognition of the original document data to determine the element position information corresponding to the original document data; data extraction of the original document data according to the element position information to obtain multi-modal element data.

[0050] Wherein, the content recognition of the original document data can be understood as the page analysis recognition of the original document data, and the content in the original document data is recognized by the page analysis technology. Actually, the content recognition of the original document data is to use the page analysis technology to locate and classify various elements in the document, including text, picture, table and noise element. The page analysis technology is used to identify different content areas in the original document data, providing a basis for subsequent content data processing. After the content recognition of the original document data, the element position information corresponding to the original document data can be determined, and the element position information is the position information of each element data in the original document data, such as the position of a certain text data in the original document data or the position of a certain picture data in the original text data. After obtaining the element position information, the multi-modal element data can be extracted from each position in the original document data according to the element position information.

[0051] In practical applications, the original document data can be of various different types, such as word, PPT, etc. In order to be able to subsequently recognize the content of the original document data, the original document data needs to be converted into PDF format in advance. All original document data of different file types can be uniformly converted into PDF format, and in order to facilitate subsequent layout analysis and recognition, the PDF format original document data can also be subjected to page processing.

[0052] In an embodiment of the present specification, the document image or PDF page of the original document data can be recognized based on a deep learning model using layout analysis technology, and the corresponding element position information can be recognized. The element position information can include text elements, image elements, table elements, title, page, footer, and other noise elements. Subsequently, the element data corresponding to each modality is extracted from the original document data according to the element position information and combined into multi-modal element data.

[0053] Based on this, through the analysis and extraction process of content recognition, element positioning, data extraction, the structured analysis of multi-complex multi-modal documents can be realized, which not only improves the accuracy of document understanding, but also provides a solid data foundation for subsequent construction of multi-modal databases and realization of intelligent retrieval and generation.

[0054] Further, data extraction is performed on the original document data according to the element position information to obtain multi-modal element data, including: determining a text position region and a non-text position region in the original document data according to the element position information; performing image recognition on the text position region and the non-text position region to obtain text element data corresponding to the text position region and non-text element data corresponding to the non-text position region; determining element link data corresponding to the non-text element data, and taking the text element data and the element link data as multi-modal element data.

[0055] The text position region can be understood as the position region corresponding to the text element, and the non-text position region can be understood as the position region corresponding to the picture element and the table element. According to the element position information, the text position region and the non-text position region in the original document data are determined, which can eliminate noise elements such as pages, footers, and blanks. Correspondingly, after image recognition is performed on the text position region and the non-text position region, the text element data corresponding to the text position region and the non-text element data corresponding to the non-text position region can be extracted. The text element data is the text content data in the original document data, and the non-text element data is the picture and table content data in the original document data.

[0056] In practical applications, image recognition of text location areas can use OCR recognition technology to convert the corresponding text image of the text location area into editable text, thereby obtaining text element data. Image recognition of non-text location areas can be processed through a visual model, thereby obtaining non-text element data. In order to process and display the non-text element data later, the non-text element data can be stored in the form of a URL link. Specifically, the picture or table can be uploaded to a server or cloud storage service, thereby obtaining element link data corresponding to the non-text element data, the element link data being a URL link, and the element link data can be saved or processed together with the text element data later.

[0057] Based on this, the determined element position information can accurately distinguish text and non-text areas from the original document data, avoiding the interference of noise elements and improving data parsing accuracy. The extracted structured data is convenient for subsequent processing, storage and retrieval. In particular, non-text elements (such as pictures and tables) are saved in the form of a URL, which is convenient for unified management and quick access. This provides a solid foundation for subsequent data management and intelligent applications.

[0058] Further, determining the element structure information corresponding to the multi-modal element data includes: performing semantic recognition on the multi-modal element data to obtain element semantic information corresponding to the multi-modal element data; and determining the element structure information corresponding to the multi-modal element data according to the element position information and the element semantic information.

[0059] The element semantic information can be understood as the logical position and hierarchical relationship of each element in the overall document structure, for example, the element is a "first-level title", a "second-level title", a "text paragraph", a "figure caption", a "table description", etc. According to the element semantic information, it can also be determined which chapter or paragraph the element belongs to in the document, for example, whether the picture belongs to the text paragraph, or whether there is a logical dependency or inclusion relationship between the front and rear elements. The element semantic information is crucial for subsequent tasks such as building a document tree, performing structured retrieval, and generating summaries.

[0060] In practical applications, in order to accurately obtain the element structure information, the element position information and the element semantic information need to be combined to determine. The element position information can determine the physical characteristics of the element such as size, page, coordinate, etc. according to the layout analysis; the element semantic information can determine the text content of the element such as "Chapter x introduction", "Figure x", etc. through the semantic recognition model; the semantic classification such as title, paragraph, list, etc.; the entity recognition such as figure x, table x, etc.; the syntax structure such as this section introduction, as shown in the figure, etc. That is, according to the element semantic information, the element content of the element can be understood. In specific implementation, the element structure information can be dynamically recognized based on the Doc-Tree algorithm by fusing the element position information and the element semantic information to determine the element structure information.

[0061] Based on this, the element structure information is determined by fusing the element position information and the element semantic information, which improves the accuracy of document structure recognition. The element structure information can clearly express the logical relationship of chapters, paragraphs, and table ownership of the document, providing a basis for subsequent construction of document trees and knowledge graphs.

[0062] Further, the element structure information corresponding to the multi-modal element data is determined according to the element position information and the element semantic information, including: analyzing the element position information and the element semantic information according to a structure matching rule, and determining the element structure information corresponding to the multi-modal element data according to the analysis result; or inputting the element position information and the element semantic information into a structure determination model to obtain the element structure information corresponding to the multi-modal element data output by the structure determination model.

[0063] The structure matching rule can be understood as a pre-defined matching rule for determining the element structure, for example, the text with a font size greater than a certain threshold and located at the beginning of a paragraph is considered as a title. The element position information and the element semantic information can be analyzed according to the structure matching rule, and the element structure information is determined according to the analysis result. The structure determination model can be understood as a model classification capability for predicting the structure of a given element. Supervised learning methods such as decision tree, random forest or deep learning model can be used, and the model can also be trained based on the Doc-Tree algorithm. The input of the structure determination model is the position information of the element and the element semantic information, and the output of the model is the element structure information corresponding to the element. Accordingly, the two methods can also be combined for use, for example, combining the advantages of rule matching and machine learning model to design a hybrid system. For highly confident cases, rules are directly applied, while for ambiguous cases, model prediction is relied on.

[0064] In practical applications, the structure matching rule is a hard logic rule based on experience or statistical rules, which is used to quickly determine the structure level of the document element, such as determining the structure according to the rules of font size, position, semantics, context, style, etc. Based on the structure matching rule, the element structure can be quickly matched and determined, but the accuracy is poor, and the adaptability to the document with inconsistent format and chaotic layout is poor. The structure determination model is a classifier based on machine learning or deep learning model, which is used to predict the structure level of the document element, and the element structure is determined by the model, which is suitable for documents with various formats and complex structures, and the training and generalization of the model improve the scene application, but it needs certain training cost and inference cost, so different ways of determining element structure information can be selected according to actual situation.

[0065] Based on this, the structure matching rule and the structure determination model can analyze the element structure information of the element based on the element position information and the element semantic information, which is convenient for subsequent arrangement of element data based on element structure information, so as to generate structured document data.

[0066] Further, the multi-modal element data is arranged based on the element structure information to obtain target document data, including: generating a document node according to the multi-modal element data; determining the node tree order corresponding to the document node by using the element structure information, and arranging the document node according to the node tree order to obtain the target document data.

[0067] Wherein, the document node can be understood as a data tree node generated according to each element data in the multi-modal element data, and the document node is used for subsequent tree structure arrangement, so as to obtain structured target document data. Therefore, by determining the node tree order of each element data corresponding to the element structure information of each element data, the document node can be arranged according to the node tree order, so as to obtain the structured target document data, so that the target document data can retain the document hierarchical structure of the original document data.

[0068] In practical applications, the node tree order refers to the logical level and arrangement order of the document node in the overall document structure, such as first-level title-second-level title, and the caption is arranged immediately after the picture node. By determining the node tree order corresponding to each document node, the document node can be arranged according to the node tree order, so as to obtain a target document data with structure retention, clear logic and convenient subsequent processing.

[0069] Based on this, by arranging the multi-modal element data using element structure information, the target document data constructed clearly retains the logical and hierarchical information of the original document, such as section structure, paragraph nesting, and chart attribution. This facilitates subsequent data block splitting processing based on the target document data, and improves the accuracy of data block splitting.

[0070] Further, the target document data is split to obtain a structured data set, including: splitting the text element data according to the element structure information to obtain at least one first text block data; splitting the at least one first text block data according to a preset block rule to obtain at least one second text block data; determining element link sub-data corresponding to the at least one second text block data in the element link data according to element position information; generating structured data based on the at least one second text block data and element link sub-data, and constructing a structured data set.

[0071] Wherein, splitting the text element data according to the element structure information can be understood as using the hierarchical information such as chapters and titles of the element structure information to preliminarily split the text element data to obtain multiple first text block data after initial splitting. Splitting the at least one first text block data according to the preset block rule can be understood as further finely splitting the first text block data according to the preset block size to avoid content fragmentation, thereby obtaining multiple second text block data after second splitting. Determining the element link sub-data corresponding to the at least one second text block data in the element link data according to the element position information can be understood as selecting the element link sub-data corresponding to each second text block data in the element link data through the coordinates in the element position information, thereby eliminating noise link data from the element link data and retaining valid link data. Generating structured data based on the second text block data and the element link sub-data can be understood as matching and combining the split second text block data with the corresponding element link, so that the structured data includes multi-modal structured data such as text, pictures, and tables, thereby realizing triple optimization of hierarchical structure retention, multi-modal association, and noise filtering.

[0072] In practical applications, the preliminary segmentation is based on the hierarchical structure information (such as chapters, titles) of the document to divide the text, with the purpose of forming first text block data with logical association. Each text block represents a relatively independent content unit, such as a first-level title and all the content below it as a block. According to the preset block splitting rule, the first text block data is finely divided. Fine-grained segmentation is further divided according to the preset block splitting rule (such as maximum character number, sentence number, etc.) on the basis of preliminary segmentation, to ensure that each second text block data is neither too large nor too small, avoiding content fragmentation. For example, the preset block splitting rule can be that each text block does not exceed 500 characters, and each text block contains no more than 3 sentences. Element link sub-data refers to the link information of non-text elements (such as pictures, tables) related to each second text block data. Through element position information (such as coordinates, context relationship), it is determined which non-text elements should be associated with the current text block, for example, assuming that a paragraph is located at the top of the page, and a picture is immediately below it, then there is an association between the paragraph and the picture, and the URL of the picture can be used as the element link sub-data of the paragraph. The second text block data and its corresponding element link sub-data are matched and combined to form the final structured data set. Each structured data entry contains not only text content, but also related pictures, tables and other multi-modal information.

[0073] Based on this, through preliminary segmentation based on hierarchical information, the chapter structure of the original document is retained, which is convenient for subsequent construction of knowledge graph or execution of structured retrieval. The fine-grained segmentation strategy can be adjusted according to actual needs (such as character number limit, paragraph number), to adapt to the needs of different application scenarios. The text block is associated with its related pictures, tables and other non-text elements, enhancing the multi-modal expression ability of the document, supporting cross-modal retrieval and generation tasks. Through element position information, irrelevant noise link data is removed, ensuring that each text block is only associated with valid non-text elements, improving data quality.

[0074] Further, determining target key data according to the structured data in the structured data set, comprising: extracting text from the structured data according to a syntax extraction rule to obtain text key data; extracting images from the structured data according to an image extraction rule to obtain image key data; extracting tables from the structured data according to a table extraction rule to obtain table key data; and generating target key data based on the text key data, the image key data, and the table key data.

[0075] The text extraction rule, the image extraction rule, and the table extraction rule can be a multi-modal extraction strategy rule based on a VLLM (Visual-Language Large Model). The text extraction of the structured data according to the syntax extraction rule can be understood as extracting text key data from text data in the structured data. The image extraction of the structured data according to the image extraction rule can be understood as extracting image key data from image data in the structured data. The table extraction of the structured data according to the table extraction rule can be understood as extracting table key data from table data in the structured data. The text key data, the image key data, and the table key data extracted can be combined as target key data.

[0076] In actual application, the text key data and the image key data can be triple data in a subject-predicate-object form, and the table key data can be data in a key-value pair form. Based on the VLLM multi-modal extraction strategy, the text and the image are extracted using the subject-predicate-object structure to ensure the structured and semantic integrity of the text information. The table extraction adopts the key-value structure to efficiently extract key information in the table and maintain the structure. In specific implementation, when text extraction is performed, NLP technology is used to identify the subject-predicate-object structure or other semantic units in a sentence, such as named entities, keywords, etc. For example, input “This experiment is guided by Professor Zhang San.”, and output the subject (Professor Zhang San), the predicate (guide), and the object (the experiment). When image extraction is performed, image recognition technology is used to identify objects and their relationships in the image and convert them into structured triple form. For example, input a landscape image, and output the subject (kitty), the predicate (lie in), and the object (under the sun). When table extraction is performed, the key-value pair form of key information is extracted by analyzing the content of the table. For example, input an experimental data summary table, and output the key (experimental date) and the value (2025-01-01).

[0077] Therefore, by using the multi-modal extraction rule to extract key data from element data of different modalities, key information can be more accurately extracted from complex documents, and semantic integrity and structure are ensured.

[0078] Further, after constructing the multi-modal database according to the target key data, the method further includes receiving a data query request for the multi-modal database, determining a query type corresponding to the data query request, determining a target sub-database in the multi-modal database based on the query type, wherein the query strategy of the target sub-database corresponds to the query type, and determining a query result corresponding to the data query request in the target sub-database according to the query strategy.

[0079] The data query request can be understood as a request for querying data proposed by a user. According to the data query request, the corresponding query type can be determined, such as text query, image query, structured query, reasoning query, etc. According to different query types, different target sub-databases in the multi-modal database can be determined. The target sub-database can be a knowledge graph sub-database, a semantic retrieval sub-database, and a collaborative document sub-database. The query strategy of different target sub-databases corresponds to the query type, that is, the knowledge graph sub-database supports complex reasoning and relationship query, the semantic retrieval sub-database supports cross-modal similarity matching, and the collaborative document sub-database supports document structured retrieval and time sequence association query. Therefore, according to the query type of the data query request, the corresponding target sub-database can be determined, and subsequent data query is performed according to the query strategy of the target sub-database.

[0080] In actual application, the multi-modal database can be divided into three different types of target sub-databases through hierarchical design, the knowledge graph sub-database corresponding to the knowledge graph layer, the semantic retrieval sub-database corresponding to the semantic retrieval layer, and the collaborative document sub-database corresponding to the collaborative document layer. According to the query type of the data query request, the request is automatically routed to the corresponding index layer, so that query is performed in the corresponding target sub-database according to the query strategy. For example, the user inputs the query "provide a photo image containing a cat sleeping", and the query type of the data query request is identified as image similarity query and structured entity matching. Then, the request can be routed to the semantic retrieval layer and the knowledge graph layer, and the knowledge graph sub-database and the semantic retrieval sub-database are used to perform query respectively using the corresponding query strategy. For example, in the knowledge graph layer, it is queried whether the photo image contains a cat sleeping, and the corresponding query data is returned according to the query result.

[0081] Based on this, through hierarchical design and dynamic routing mechanism, the bottleneck problem of single index structure in scalability, efficiency and multi-modal support is solved.

[0082] The specification provides a multi-modal database construction method based on a visual language model collaborative routing, which includes parsing original document data to obtain multi-modal element data, and determining element structure information corresponding to the multi-modal element data; arranging the multi-modal element data based on the element structure information to obtain target document data; splitting the target document data to obtain a structured data set, and determining target key data according to structured data in the structured data set; and constructing a multi-modal database according to the target key data, wherein the multi-modal database is used for data retrieval. The original document data is parsed to obtain multi-modal element data, so that the original document data can be more comprehensively processed subsequently. The element structure information of the multi-modal element data is determined, and the multi-modal element data is arranged based on the element structure information, so that the target document data obtained can retain the structure information of the original document data, facilitating subsequent splitting of the target document data. The target document data is split to obtain a structured data set, which can obtain more detailed structured data and ensure data integrity, avoiding data fragmentation problems. The target key data is determined according to the structured data in the structured data set, and the multi-modal database is constructed according to the target key data, realizing efficient processing of multi-modal unstructured data, improving the database construction efficiency, and at the same time, reducing the retrieval time and improving the retrieval efficiency when data retrieval is performed based on the multi-modal database subsequently.

[0083] The following description is combined with the accompanying Figure 3 The multi-modal database construction method based on the visual language model collaborative routing provided by the specification is further described by taking the application of the multi-modal database construction method based on the visual language model collaborative routing in data construction as an example. Wherein, Figure 3 A processing process flowchart of a multi-modal database construction method based on a visual language model collaborative routing provided by an embodiment of the specification is shown, which specifically includes the following steps.

[0084] Step 302: Content recognition is performed on the original document data to determine element position information corresponding to the original document data; and data extraction is performed on the original document data according to the element position information to obtain multi-modal element data.

[0085] In an implementable manner, the original document data is document data to be imported into a database, content of the original document data is recognized by using a layout analysis technology, element position information corresponding to each element in the original document data is determined, a text position region and a non-text position region in the original document data are determined according to the element position information; image recognition is performed on the text position region and the non-text position region, text element data corresponding to the text position region and non-text element data corresponding to the non-text position region are obtained; element link data corresponding to the non-text element data is determined, and the text element data and the element link data are used as multi-modal element data.

[0086] Step 304: performing semantic recognition on the multi-modal element data to obtain element semantic information corresponding to the multi-modal element data; and determining element structure information corresponding to the multi-modal element data according to the element position information and the element semantic information.

[0087] In an implementable manner, the semantic recognition model is used to perform semantic recognition on the multi-modal element data to obtain element semantic information corresponding to the multi-modal element data, the element position information and the element semantic information are analyzed by using the structure matching rule, and element structure information corresponding to the multi-modal element data is determined according to an analysis result; or the element position information and the element semantic information are input into a structure determination model to obtain element structure information corresponding to the multi-modal element data output by the structure determination model.

[0088] Step 306: generating a document node according to the multi-modal element data; determining a node tree sequence corresponding to the document node by using the element structure information, arranging the document node according to the node tree sequence, and obtaining target document data.

[0089] Step 308: splitting the text element data according to the element structure information to obtain at least one first text block data; and splitting the at least one first text block data according to a preset block rule to obtain at least one second text block data.

[0090] Step 310: determining element link sub-data corresponding to the at least one second text block data in the element link data according to the element position information; generating structured data and constructing a structured data set based on the at least one second text block data and the element link sub-data.

[0091] Step 312: determining target key data according to the structured data in the structured data set.

[0092] In an implementable manner, text key data is obtained by performing text extraction on the structured data according to a text extraction rule; image key data is obtained by performing image extraction on the structured data according to an image extraction rule; table key data is obtained by performing table extraction on the structured data according to a table extraction rule; and target key data is generated based on the text key data, the image key data, and the table key data.

[0093] The multi-modal database construction method based on the visual language model collaborative routing provided in the specification realizes parsing of original document data to obtain multi-modal element data, so that subsequent processing of the original document data can be more comprehensive. The element structure information of the multi-modal element data is determined, and the multi-modal element data is arranged based on the element structure information, so that the target document data obtained can retain the structure information of the original document data, facilitating subsequent segmentation of the target document data. The structured data set is obtained by segmenting the target document data, which can obtain more detailed structured data and ensure data integrity, avoiding data fragmentation problems. The target key data is determined according to the structured data in the structured data set, and the multi-modal database is constructed according to the target key data, realizing efficient processing of multi-modal unstructured data, improving the database construction efficiency, and at the same time, reducing the search time and improving the search efficiency when subsequent data search is performed based on the multi-modal database.

[0094] Corresponding to the method embodiments described above, the specification also provides a multi-modal database construction device based on the visual language model collaborative routing, Figure 4 A structural schematic diagram of a multi-modal database construction device based on the visual language model collaborative routing is shown, which is provided by an embodiment of the specification. As shown in Figure 4 The device comprises:

[0095] The analysis module 402 is configured to parse the original document data to obtain multi-modal element data, and determine element structure information corresponding to the multi-modal element data;

[0096] The arrangement module 404 is configured to arrange the multi-modal element data based on the element structure information to obtain target document data;

[0097] The segmentation module 406 is configured to segment the target document data to obtain a structured data set, and determine target key data according to structured data in the structured data set;

[0098] The adding module 408 is configured to add the target key data to a multi-modal database, wherein the multi-modal database is used for data search.

[0099] Optionally, the parsing module 402 is further configured to perform content recognition on the original document data to determine element position information corresponding to the original document data; and perform data extraction on the original document data according to the element position information to obtain multi-modal element data.

[0100] Optionally, the parsing module 402 is further configured to determine a text position region and a non-text position region in the original document data according to the element position information; perform image recognition on the text position region and the non-text position region to obtain text element data corresponding to the text position region and non-text element data corresponding to the non-text position region; and determine element link data corresponding to the non-text element data, and take the text element data and the element link data as the multi-modal element data.

[0101] Optionally, the parsing module 402 is further configured to perform semantic recognition on the multi-modal element data to obtain element semantic information corresponding to the multi-modal element data; and determine element structure information corresponding to the multi-modal element data according to the element position information and the element semantic information.

[0102] Optionally, the parsing module 402 is further configured to analyze the element position information and the element semantic information according to a structure matching rule, determine element structure information corresponding to the multi-modal element data according to an analysis result, or input the element position information and the element semantic information into a structure determination model to obtain element structure information corresponding to the multi-modal element data output by the structure determination model.

[0103] Optionally, the parsing module 402 is further configured to generate a document node according to the multi-modal element data; determine a node tree order corresponding to the document node by using the element structure information; arrange the document node according to the node tree order to obtain target document data.

[0104] Optionally, the segmentation module 406 is further configured to segment the text element data according to the element structure information to obtain at least one first text block data; segment the at least one first text block data according to a preset block rule to obtain at least one second text block data; determine element link sub-data corresponding to the at least one second text block data in the element link data according to the element position information; and generate structured data and construct a structured data set based on the at least one second text block data and the element link sub-data.

[0105] Optionally, the segmentation module 406 is further configured to perform text extraction on the structured data according to a syntax extraction rule to obtain text key data, perform image extraction on the structured data according to an image extraction rule to obtain image key data, and perform table extraction on the structured data according to a table extraction rule to obtain table key data, and generate target key data based on the text key data, the image key data, and the table key data.

[0106] Optionally, the apparatus further includes a query module configured to receive a data query request for the multi-modal database, determine a query type corresponding to the data query request, determine a target sub-database in the multi-modal database based on the query type, wherein a query strategy of the target sub-database corresponds to the query type, and determine a query result corresponding to the data query request in the target sub-database according to the query strategy.

[0107] The above is a schematic scheme of the multi-modal database construction apparatus based on the visual language model collaborative routing according to an embodiment. It should be noted that the technical scheme of the multi-modal database construction apparatus based on the visual language model collaborative routing is the same as the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing, and the details of the technical scheme of the multi-modal database construction apparatus based on the visual language model collaborative routing that are not described in detail can be seen from the description of the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing.

[0108] Figure 5 A structural block diagram of a computing device 500 is shown according to an embodiment of the present specification. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected with the memory 510 through a bus 530, and a database 550 is used to save data.

[0109] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or combinations of such networks, such as the Internet. The access device 540 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as a wired or wireless network interface, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC).

[0110] In one embodiment of the present specification, the above-mentioned components of the computing device 500 and other components not shown in the Figure 5 may be connected to each other, such as through a bus. It should be understood that Figure 5 The computing device structure diagram shown is only for the purpose of example, and is not a limitation on the scope of the present specification. Those skilled in the art can add or replace other components as needed.

[0111] The computing device 500 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (for example, a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, and the like), a mobile phone (for example, a smartphone), a wearable computing device (for example, a smart watch, smart glasses, and the like), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 can also be a mobile or stationary server.

[0112] The processor 520 is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned method for constructing a multi-modal database based on a visual language model collaborative routing.

[0113] The above is a schematic scheme of the computing device of the embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing described above belong to the same concept, and the details of the technical scheme of the computing device that are not described in detail can be referred to the description of the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing.

[0114] An embodiment of the present specification also provides a computer readable storage medium storing computer executable instructions, which, when executed by a processor, implement the steps of the multi-modal database construction method based on the visual language model collaborative routing described above.

[0115] The above is a schematic scheme of the computer readable storage medium of the embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing described above belong to the same concept, and the details of the technical scheme of the storage medium that are not described in detail can be referred to the description of the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing.

[0116] An embodiment of the present specification also provides a computer program product comprising a computer program or instructions, which, when executed by a processor, implement the steps of the multi-modal database construction method based on the visual language model collaborative routing described above.

[0117] The above is a schematic scheme of the computer program product of the embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing described above belong to the same concept, and the details of the technical scheme of the computer program product that are not described in detail can be referred to the description of the technical scheme of the multi-modal database construction method based on the visual language model collaborative routing.

[0118] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in an order other than that described in the embodiments and still achieve desirable results. Additionally, the process depicted in the figures does not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or necessary.

[0119] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc. It should be noted that the computer readable medium can include appropriate additions or subtractions according to the requirements of patent practice. For example, according to the patent practice in some regions, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0120] It should be noted that, for the foregoing method embodiments, in order to facilitate description, each is described as a combination of a series of acts, but those skilled in the art should appreciate that the embodiments of the present specification are not limited by the order of the described acts, because according to the embodiments of the present specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should appreciate that the embodiments described in the specification are all preferred embodiments, and the acts and modules involved are not necessarily essential to the embodiments of the present specification.

[0121] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0122] The preferred embodiments of the present specification disclosed above are only used to help explain the present specification. The alternative embodiments do not describe all the details and do not limit the invention to the specific embodiments described. Obviously, according to the content of the embodiments of the present specification, many modifications and changes can be made. The present specification selects and describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present specification, so that those skilled in the art can well understand and use the present specification.

Claims

1. A method for constructing a multi-modal database based on visual language model collaborative routing, characterized in that, The method comprises the following steps: parsing original document data to obtain multi-modal element data, and determining element structure information corresponding to the multi-modal element data; The parsing of the original document data comprises: determining element position information by performing content recognition on the original document data, and extracting multi-modal element data from the original document data according to the element position information, wherein the multi-modal element data comprises text element data and non-text element data, and the non-text element data corresponds to element link data for storage; the determination of the element structure information corresponding to the multi-modal element data comprises: obtaining element semantic information by performing semantic recognition on the multi-modal element data, and determining the element structure information corresponding to the multi-modal element data according to the element position information and the element semantic information; arranging the multi-modal element data based on the element structure information to obtain target document data; the target document data comprises document nodes with a node tree sequence, and the document nodes are obtained by generating the multi-modal element data; segmenting the target document data to obtain a structured data set, and determining target key data according to structured data in the structured data set; the segmentation comprises preliminary segmentation based on hierarchical information and fine-grained segmentation based on a preset block size; the determination of the target key data according to the structured data in the structured data set comprises: multi-modal extraction of structured data in the structured data based on a visual language model to determine the target key data, wherein the text data and the image data are extracted using a subject-predicate-object structure, and the table data is extracted using a key-value structure; constructing a multi-modal database according to the target key data, wherein the multi-modal database is used for data retrieval according to an index mechanism comprising a knowledge graph layer, a semantic retrieval layer and a collaborative document layer.

2. The method of claim 1, wherein, extracting multi-modal element data from the original document data according to the element position information, comprising: determining a text position area and a non-text position area in the original document data according to the element position information; performing image recognition on the text position area and the non-text position area to obtain text element data corresponding to the text position area and non-text element data corresponding to the non-text position area; determining element link data corresponding to the non-text element data, and taking the text element data and the element link data as multi-modal element data.

3. The method of claim 1, wherein, determining element structure information corresponding to the multi-modal element data according to the element position information and the element semantic information, comprising: analyzing the element position information and the element semantic information according to a structure matching rule, and determining the element structure information corresponding to the multi-modal element data according to an analysis result; or inputting the element position information and the element semantic information into a structure determination model to obtain the element structure information corresponding to the multi-modal element data output by the structure determination model.

4. The method of claim 1, wherein, arranging the multi-modal element data based on the element structure information to obtain target document data, comprising: generating a document node according to the multi-modal element data; determining a node tree order corresponding to the document node according to the element structure information, and arranging the document node according to the node tree order to obtain target document data.

5. The method of claim 1, wherein, segmenting the target document data to obtain a structured data set, including: segmenting the text element data according to the element structure information to obtain at least one first text block data; segmenting the at least one first text block data according to a preset segmentation rule to obtain at least one second text block data; determining element link sub-data corresponding to the at least one second text block data in the element link data according to the element position information; generating structured data and constructing a structured data set based on the at least one second text block data and the element link sub-data.

6. The method of claim 5, wherein, determining target key data according to the structured data in the structured data set, including: extracting text key data from the structured data according to a syntax extraction rule; extracting image key data from the structured data according to an image extraction rule; extracting table key data from the structured data according to a table extraction rule; generating target key data based on the text key data, the image key data, and the table key data.

7. The method according to any one of claims 1 to 6, characterized in that, After constructing the multi-modal database according to the target key data, further including: receiving a data query request for the multi-modal database, and determining a query type corresponding to the data query request; determining a target sub-database in the multi-modal database based on the query type, wherein a query strategy of the target sub-database corresponds to the query type; determining a query result corresponding to the data query request in the target sub-database according to the query strategy. 8.A multi-modal database construction device based on visual language model collaborative routing, characterized in that, including: a parsing module configured to parse original document data to obtain multi-modal element data, and determine element structure information corresponding to the multi-modal element data; the parsing original document data includes: determining element position information by content recognition on the original document data, and extracting multi-modal element data according to the element position information, wherein the multi-modal element data includes text element data and non-text element data, and the non-text element data corresponds to element link data for storage; the determining element structure information corresponding to the multi-modal element data includes: obtaining element semantic information by semantic recognition on the multi-modal element data, and determining element structure information corresponding to the multi-modal element data according to the element position information and the element semantic information; an arrangement module configured to arrange the multi-modal element data based on the element structure information to obtain target document data; the target document data includes document nodes with a node tree order, and the document nodes are obtained according to the multi-modal element data; The segmentation module is configured to segment the target document data to obtain a structured data set, and determine target key data according to structured data in the structured data set; the segmentation includes preliminary segmentation based on hierarchical information and fine-grained segmentation based on a preset block size; the determination of the target key data according to the structured data in the structured data set includes: multi-modal extraction of structured data in the structured data based on a visual language model to determine the target key data, wherein text data and image data are extracted using a subject-predicate-object structure, and table data is extracted using a key-value structure; The adding module is configured to add the target key data to a multi-modal database, wherein the multi-modal database is used for data retrieval according to an index mechanism composed of a knowledge graph layer, a semantic retrieval layer and a collaborative document layer.

9. A computing device, comprising: Comprise: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions, and the computer executable instructions realize the steps of the multi-modal database construction method based on the visual language model collaborative routing according to any one of claims 1 to 7 when executed by the processor.

10. A computer-readable storage medium, characterized in that, which stores computer executable instructions, which realize the steps of the multi-modal database construction method based on the visual language model collaborative routing according to any one of claims 1 to 7 when executed by the processor.

11. A computer program product, characterised in that, comprise computer programs or instructions, which realize the steps of the multi-modal database construction method based on the visual language model collaborative routing according to any one of claims 1 to 7 when executed by the processor.

Images