A multimodal retrieval method and apparatus

Abstract

The application discloses a multimodal retrieval method and device, relates to the technical field of multimodal retrieval, and comprises the following steps: obtaining a query vector corresponding to a user query text, then querying the vector database according to the query vector to generate a query result; the vector database stores vector encodings corresponding to a plurality of multi-level content block structures of a plurality of multimodal table documents; the multi-level content block structure of any multimodal table document comprises an atomic layer, a semantic layer and a relationship layer; the atomic block comprises target element content and structure information of any cell in the multimodal table document; the semantic block comprises aggregated content corresponding to any row, any column or any whole table in the multimodal table document; the relationship block comprises text describing the association between a plurality of atomic blocks and a plurality of semantic blocks; and the structure information comprises row and column indexes corresponding to each cell and cross-modal association between multimodal elements. The application can improve the accuracy of multimodal retrieval.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal retrieval, in particular to a multi-modal retrieval method and device. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) technology effectively reduces the "hallucination" phenomenon in generated answers by combining information retrieval with large language models. Traditional RAG is based on embedding technology, which retrieves unstructured text information through semantic similarity matching, and has good performance.

[0003] However, the table data in actual enterprise applications often contains structured text, images, formulas and other multi-modal content, so the above traditional RAG method focuses on the extraction of text-type cell content, and thus ignores the extraction of other modal information, resulting in low retrieval accuracy. At the same time, since the structured data in the table is often organized according to a specific structure and hierarchy, there is no fixed order or common rule in natural language, and thus subsequent queries cannot be quickly retrieved through structural positioning, but only fuzzy queries, which is inefficient and prone to data omission.

[0004] Therefore, how to improve the accuracy of multi-modal retrieval has become a problem to be solved. SUMMARY

[0005] The present application provides a multi-modal retrieval method and device to at least solve the problem of low accuracy of multi-modal retrieval in related technologies.

[0006] The present application provides a multi-modal retrieval method, comprising:

[0007] receiving a user query text;

[0008] vector encoding the user query text to obtain a query vector corresponding to the user query text;

[0009] querying according to the query vector in a vector database to generate a query result; the vector database stores vector encodings corresponding to multi-level content block structures of a plurality of multi-modal table documents;

[0010] The multi-level content block structure of any of the multi-modal table documents comprises an atomic layer composed of a plurality of atomic blocks, a semantic layer composed of a plurality of semantic blocks, and a relationship layer composed of a plurality of relationship blocks; the atomic block comprises target element content and structure information of any cell in the multi-modal table document; the semantic block comprises aggregated content corresponding to any row, or any column, or any whole table in the multi-modal table document; the relationship block comprises text describing the association between a plurality of the atomic blocks and a plurality of the semantic blocks; and the structure information comprises row and column indexes corresponding to each cell, and cross-modal association between multi-modal elements.

[0011] The application further provides a multi-modal retrieval device, comprising:

[0012] a receiving unit configured to receive a user query text;

[0013] an encoding unit configured to perform vector encoding on the user query text to obtain a query vector corresponding to the user query text;

[0014] a querying unit configured to perform querying in a vector database according to the query vector to generate a query result; the vector database stores vector encodings corresponding to multi-level content block structures of a plurality of multi-modal table documents;

[0015] The multi-level content block structure of any of the multi-modal table documents comprises an atomic layer composed of a plurality of atomic blocks, a semantic layer composed of a plurality of semantic blocks, and a relationship layer composed of a plurality of relationship blocks; the atomic block comprises target element content and structure information of any cell in the multi-modal table document; the semantic block comprises aggregated content corresponding to any row, or any column, or any whole table in the multi-modal table document; the relationship block comprises text describing the association between a plurality of the atomic blocks and a plurality of the semantic blocks; and the structure information comprises row and column indexes corresponding to each cell, and cross-modal association between multi-modal elements.

[0016] The application further provides an electronic device, comprising a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the multi-modal retrieval methods.

[0017] The application further provides a computer-readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the multi-modal retrieval methods.

[0018] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of any of the multi-modal retrieval methods.

[0019] The present application can completely cover the content of each cell, the aggregation semantics of rows and columns of the whole table, and the cross-element association logic in the multi-modal table by constructing a multi-level content block structure including an atomic layer (corresponding to single-cell multi-modal elements and structure information), a semantic layer (corresponding to the aggregation content of rows, columns or the whole table), and a relationship layer (describing the association relationship between atomic blocks and semantic blocks), realizing integrated retrieval of multi-modal elements and structures and associations to adapt to complex scenarios. With the representation of each level of content block by vector encoding, the user query vector can be accurately matched with the vector of the atomic block, the semantic block and the relationship block in the vector database, covering multi-dimensional retrieval requirements such as cell details and row / column aggregation semantics, breaking through the limitation of traditional table retrieval focusing only on the content itself through the separately set “relationship layer”, supporting deep retrieval based on association logic, reducing missed and false detections, and significantly improving semantic matching accuracy. At the same time, the combination of multi-level structure and vector retrieval can also flexibly adapt to user query intentions from fine-grained details to medium-grained aggregation and then to association logic, greatly improving the flexibility of retrieval and user experience. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 One of the flowcharts of a multi-modal retrieval method provided by an embodiment of the present application;

[0022] Figure 2 The flowchart of a multi-modal retrieval method provided by an embodiment of the present application;

[0023] Figure 3 The flowchart of a multi-modal retrieval method provided by an embodiment of the present application;

[0024] Figure 4 The flowchart of a multi-modal retrieval method provided by an embodiment of the present application;

[0025] Figure 5 The structural diagram of a multi-modal retrieval device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] This application provides a multimodal retrieval method, and the method is described in detail below with reference to the execution flow of the multimodal retrieval method. Figure 1 The diagram shown is a flowchart of the multimodal retrieval method provided in this application. The specific steps include the following:

[0030] S101, Receive user query text.

[0031] In some embodiments, the received user query text may be natural language query text entered by the user (e.g., "What was the profit of product A in Q3 2024?"), and then, in conjunction with the actual needs corresponding to the user query text, a query is performed using a vector database and / or a relational database to generate query results.

[0032] S102. Perform vector encoding on the user query text to obtain the query vector corresponding to the user query text.

[0033] Furthermore, the user's query text needs to be vector-encoded, transforming the query intent in text form into a computer-computable vector form, namely the query vector. This allows the matching degree between vectors to be measured in the vector database through vector similarity. Finally, the query is performed in the vector database based on the query vector to generate the results.

[0034] S103. Perform a query in the vector database based on the query vector and generate the query results.

[0035] The vector database stores vector encodings corresponding to multi-level content block structures of a plurality of multi-modal table documents.

[0036] Specifically, the multi-level content block structure of any multi-modal table document includes an atomic layer composed of a plurality of atomic blocks, a semantic layer composed of a plurality of semantic blocks, and a relationship layer composed of a plurality of relationship blocks; the atomic block includes target element content and structure information of any cell in the multi-modal table document; the semantic block includes aggregated content corresponding to any row, or any column, or any entire table in the multi-modal table document; the relationship block includes text describing the association between the plurality of atomic blocks and the plurality of semantic blocks; the structure information includes row and column indexes corresponding to each cell, and cross-modal association between multi-modal elements.

[0037] It should be noted that the vector database in the embodiment of the present application pre-stores vector encodings of multi-level content block structures of a large number of multi-modal tables that may be used, such as atomic layer (single cell content and structure), semantic layer (row / column / entire table aggregated content), and relationship layer (association between content blocks). Therefore, based on the three dimensions of the content of each cell, the aggregated semantics of the row, column, and entire table, and the cross-element association logic, the vector can be accurately matched, and the content most suitable for the user's query intention can be searched, and finally a comprehensive and accurate retrieval result can be generated.

[0038] Specifically, the multi-level content block structure includes three layers, specifically including the following:

[0039] (1) Atomic layer

[0040] The atomic layer includes a plurality of atomic blocks. The atomic block can be understood as the smallest carrier of fine-grained information. In units of single cells, the target element content and structure information of each cell are associated and bound to form an independent atomic block.

[0041] For example: Atomic block 1 = [target element content (Markdown format "Product A") + structure information (row 3 column 2 index + no cross-modal association)].

[0042] For example: Atomic block 2 = [target element content (LaTeX format "Profit = Sales × (Unit Price - Cost)") + structure information (row 5 column 4 index + associated row 5 column 2 "Sales", row 5 column 3 "Unit Price" atomic block)].

[0043] By constructing an atomic block for each cell, an atomic layer is formed, thereby preserving the fine-grained information integrity of the table. Each atomic block is an independent unit that can be positioned and includes its semantic information, which is the basis for subsequent aggregation and association. At the same time, when querying, if the information of a specific cell needs to be accurately positioned, the atomic block can be directly matched quickly.

[0044] (2) Semantic layer

[0045] The semantic layer includes a plurality of semantic blocks. The semantic blocks are formed by aggregating the atomic blocks in three dimensions of row, column, and whole table based on the row index and column index in the structure information, and the association relationship between the corresponding atomic blocks is retained.

[0046] It should be noted that aggregation refers to the process of integrating and summarizing a plurality of dispersed and fine-grained information units (e.g., atomic blocks) into a more holistic semantic information set (i.e., semantic block) according to a specific rule (e.g., the structural logic of row, column, and whole table).

[0047] Among them, the row-level semantic block aggregates all atomic blocks in the same row, for example: row 3 semantic block = atomic block set of row 3 column 1-column 5, and thus can describe the complete semantics of a row. When the data of a certain row is for a plurality of cells of product A, the semantic information corresponding to the row-level semantic block corresponding to the row can include "name, unit price, sales, cost, and profit of product A".

[0048] The column-level semantic block aggregates all atomic blocks in the same column, for example: column 2 semantic block = atomic block set of column 2 row 1-row 10, and thus can describe the dimension semantics of a column, such as the unit price data of all products.

[0049] The table-level semantic block aggregates the row-level and column-level semantic blocks of the whole table, and records the total range of the rows and columns of the table, for example, "row 1-row 10, column 1-column 5", which can describe the global semantics of the whole table, such as the product sales data table of Q3 (third quarter) of 2024.

[0050] (3) Relationship layer

[0051] The relationship layer includes a plurality of relationship blocks. The relationship blocks are formed by describing the association logic between atomic blocks and semantic blocks using natural language text based on the cross-modal association relationship in the structure information.

[0052] For example: relationship block 1 = [atomic block (row 5 column 4 formula) references atomic block (row 5 column 2 sales), atomic block (row 5 column 3 unit price) - association type: data dependency].

[0053] For another example: relationship block 2 = [row 3 semantic block (product A data) is associated with column 5 semantic block (all product profit data) - association type: data attribution].

[0054] The association logic of multi-modal elements is established through the relationship blocks in the relationship layer, so that the retrieval can extend from a single block to associated blocks (for example, when searching for a formula, the original data cell referenced by the formula can be traced back).

[0055] Specifically, the construction process of the multi-level content block structure includes the following steps 1 to 3:

[0056] Step 1, parse a plurality of multi-modal table documents to obtain target element content and structure information corresponding to each cell in each multi-modal table document.

[0057] The structure information includes the row and column indexes corresponding to each cell, and the cross-modal association relationship between multi-modal elements.

[0058] Specifically, the multi-modal table document refers to a table whose multiple cells contain multiple data types (modalities), for example, the multi-modal table document can be an Excel format table document; the multi-modal can include text, pictures, formulas, and other different modalities; the text element can be understood as ordinary numbers and text content; the picture element is an icon, diagram, photo, or the like embedded in the cell; the formula element is a mathematical formula, chemical equation, or the like. For example, the multi-modal table document can be a data table in a scientific research report, a corporate financial statement, a product parameter table, or the like.

[0059] It should be noted that, due to the non-uniform format of multi-modal elements in the multi-modal table document, for example, the text element can carry differentiated formats such as bold, line break, and special symbol coding, the picture element can be embedded in different formats such as JPG and PNG without textual description, and the formula element can be a "picture formula" in the form of a screenshot or rely on a specific software (such as Math Type) for exclusive coding. Directly preserving the original format will lead to difficulties in identifying the same content and associating non-text elements during subsequent parsing; therefore, the elements in the multi-modal table document need to be converted in format, and ultimately the target element content format is unified and semantically, laying a foundation for subsequent construction of multi-level content blocks.

[0060] Therefore, the data format is unified based on the following steps 11-14 in the embodiments of the present application to obtain the target element content corresponding to each cell:

[0061] Step 11, parse a plurality of multi-modal table documents to extract initial element content in each cell in each multi-modal table document.

[0062] The initial element content includes at least one of a text element, a picture element, and a formula element.

[0063] Specifically, the initial element content in each cell, i.e., the text element, the picture element, and the formula element, can be extracted from the plurality of multi-modal table documents by using a table parsing tool (such as openpyxl for Excel, Tabula for PDF, etc.).

[0064] Step 12, when the initial element content is a text element, convert the format of the text element into the target lightweight markup language format to generate the corresponding target element content.

[0065] In some embodiments, the target lightweight markup language format can be Markdown, which converts raw text from different software with chaotic formats into a unified, concise, and parsable standard format, eliminating format interference while preserving key semantics and necessary format information. For example, use "\n" to represent line breaks, thereby eliminating format redundancy and differences, focusing on core semantics, and facilitating subsequent vector encoding and semantic matching.

[0066] Step 13, when the initial element content is a picture element, perform semantic understanding on the picture element to generate a corresponding textual description of the picture element, thereby generating the corresponding target element content.

[0067] Specifically, through an image understanding model, such as a CLIP-based image-text association model, a textual description of icons, photos, etc. within the cell is generated, for example, the textual description of the picture can be "male, dark short hair, wearing a blue shirt, background is white", converting visual information that cannot directly participate in semantic calculation into text information, enabling the picture to establish semantic association with other elements (e.g., corresponding text data).

[0068] Step 14, when the initial element content is a formula element, convert the formula element into a text in the target syntax format to generate the corresponding target element content.

[0069] Wherein, the target syntax format can be LaTeX structured syntax text, thereby converting the formula into LaTeX structured syntax format, preserving the mathematical logic of the formula and achieving cross-platform compatibility, facilitating subsequent parsing of formula meaning and association with referenced cell data.

[0070] The embodiments of the present application convert text into a unified lightweight markup language format and formulas into standardized target syntax text, while activating the semantic associability of non-text elements such as pictures and formulas, laying a high-quality data foundation for subsequent extraction of structural information, construction of multi-level content blocks, and database retrieval.

[0071] It should be further noted that structural information is information that defines the spatial position of elements and the association relationship between elements in a multi-modal table document, integrating scattered multi-modal elements into a logical and traceable whole.

[0072] Specifically, the generation process of the structural information can refer to the following steps:

[0073] For any of the plurality of multi-modal table documents, the row and column indexes of each cell in the multi-modal table document file, and the cross-modal association relationship between the text elements, picture elements and formula elements corresponding to each cell in the multi-modal table document are extracted to generate structure information.

[0074] Among them, the row and column indexes can clearly indicate the specific coordinates of each cell in the table (for example, row 3 column 2), providing a unique spatial identifier for all multi-modal elements; the cross-modal association relationship between the text elements, picture elements and formula elements corresponding to each cell can be understood as the logical reference relationship between different modal elements, and then the semantic association logic of multi-modal elements is established. This lays a foundation for subsequent sorting of association logic between atomic blocks and semantic blocks, and for association tracing query when querying.

[0075] Step 2, for each multi-modal table document, based on the target element content corresponding to each cell and the structure information, a multi-level content block structure corresponding to the multi-modal table document is constructed.

[0076] Further, the multi-level content block structure corresponding to each multi-modal table document is constructed according to the target element content corresponding to each cell and the structure information of each multi-modal table document obtained in step 1.

[0077] Specifically, the multi-level content block structure constructed can refer to the description in the above step S103, which will not be repeated here.

[0078] Step 3, vector encoding is performed on the multi-level content blocks, and the vector database is stored. In this step, the atomic blocks, semantic blocks, and relationship blocks, i.e., the multi-level content block structure constructed in the foregoing are vector encoded. Specifically, an embedded model (such as BERT, CLIP, etc.) can be used to convert the "textual information" (such as Markdown text of atomic blocks, textual description of pictures, LaTeX text of formulas, and association description of relationship blocks) in the atomic blocks, semantic blocks, and relationship blocks into high-dimensional feature vectors. These vectors can accurately capture the semantic features of the content (for example, the vectors of "product A sales" and "product A sales" will be highly similar); the generated vectors and corresponding "block identifiers" (such as "atomic block-table 1-row 3-column 2" and "row-level semantic block-table 1-row 5") are stored in a vector database (such as Milvus, Chroma, etc.).

[0079] Further, when the user inputs a natural language query text (such as "find product A sales data"), the system can also convert the query text into a vector, calculate the similarity between the query vector and the content block vector stored in the vector database, and quickly locate the semantically associated content block without relying on accurate keywords or formats.

[0080] The present application can completely cover the content of each cell, the aggregation semantics of the row and column of the whole table, and the cross-element association logic in the multi-modal table by constructing a multi-level content block structure including an atomic layer (corresponding to a single cell multi-modal element and structure information), a semantic layer (corresponding to the aggregation content of a row, a column or a whole table), and a relationship layer (describing the association relationship between atomic blocks and semantic blocks), realizing integrated retrieval of multi-modal elements and structures, and association to adapt to complex scenarios. With the representation of the content blocks at each level by vector encoding, the user query vector can be accurately matched with the vectors of the atomic blocks, the semantic blocks and the relationship blocks in the vector database, covering multi-dimensional retrieval requirements such as cell details and row / column aggregation semantics, breaking through the limitation of traditional table retrieval which only focuses on the content itself through the separately set “relationship layer”, supporting deep retrieval based on association logic, reducing missed and false detections, and significantly improving the semantic matching accuracy. At the same time, the combination of the multi-level structure and vector retrieval can also flexibly adapt to the full-scenario user query intent from fine-grained details to medium-grained aggregation and then to association logic, greatly improving the retrieval flexibility and user experience.

[0081] As an extension and refinement of the above-mentioned embodiments, referring to FIG. 8, the present application also provides another multi-modal retrieval method, and the specific steps include the following: Figure 2

[0082] S201, receiving a user query text.

[0083] S202, performing vector encoding on the user query text to obtain a query vector corresponding to the user query text.

[0084] S203, inputting the user query text into a target semantic recognition model to perform intent recognition on the semantics of the user query text by the target semantic recognition model, and obtaining an intent recognition result.

[0085] Specifically, the user query text (for example, “find cost accounting formula” “statistical product AQ3 profit”) can be input into the target semantic recognition model (usually a text classification / intent recognition model based on BERT / LLaMA), and then the intent recognition result of the user query text output by the model is obtained. Specifically, the intent recognition result of the user can be divided into two categories, namely, a fuzzy semantic intent and a refined semantic intent. The fuzzy semantic intent refers to no explicit data condition / calculation requirement, and focuses on semantic association matching (for example, “which content and sales trend graph are related” “what are the formulas for cost accounting”); the refined semantic intent refers to containing explicit data conditions, calculation targets or location information, and focusing on “precise data extraction / operation” (for example, “products with profit > 100,000 in table 1” “total sales of product A in Q3 of 2024”).

[0086] S204, querying the vector database and the relational database according to the intent recognition result and the query vector to generate a query result.​

[0087] wherein the structured content of the plurality of multi-modal table documents is saved in the relational database.

[0088] In the embodiments of the present application, information with explicit structured features in the multi-modal table also needs to be extracted, which can include: table basic information, i.e., table ID, table name, total number of rows / columns; cell structured data, i.e., row / column index of each cell, target element content (e.g., normalized pure text, formula calculation result, picture description keyword), belonging table ID; content block association information, i.e., the attribution relationship between atomic blocks and semantic blocks (e.g., "atomic block-table 1-row 3-column 2 belongs to row-level semantic block-table 1-row 3"), association identification and type of both parties recorded by the relationship block.

[0089] Further, the above structured information is stored in a relational database (e.g., using MySQL, PostgreSQL) according to the structure of "table-row-column-cell-content block".

[0090] The SQL statement is used to realize conditional filtering (e.g., "filtering products with unit price > 20 yuan in table 1"), "data aggregation" (e.g., "statistical total sales of all products in table 1"), "association query" (e.g., "query all atomic blocks under row 3 semantic block"), to ensure efficient response to "precise data operation requirements".

[0091] Then, the characteristics of both the vector database and the relational database can be combined, the relational database can make up for the short board of insufficient semantic matching precision of the vector database, and the vector database can solve the limitation that traditional retrieval needs to match precise keywords. The combination of the two can cover various needs of users from fuzzy questioning to precise data calculation.

[0092] Specifically, referring to FIG. 2, the detailed steps of the step S204 (querying according to the intention recognition result and the query vector, combining the vector database and the relational database to generate a query result) include the following: Figure 3

[0093] S2041, judging whether the intention recognition result contains a target requirement.

[0094] The target requirement is any one of the data query requirements with precision in data aggregation, conditional filtering, and multi-table association requirements.

[0095] In the above S2041, if the intention recognition result contains the target requirement, the following S2042 is executed, and if the intention recognition result does not contain the target requirement, the following S2043 is executed:

[0096] ​S2042, query the vector database based on the query vector to obtain a first data query result, and obtain a structured query statement corresponding to the user query text, query the relational database through the structured query statement to obtain a second data query result; and integrate the first data query result and the second data query result to generate a query result.

[0097] When the intent recognition result is a query with both fuzzy semantic requirements and precise requirements, for example, "How much is the profit of product A in Q3 of 2024?", it indicates that the current user requirements contain explicit precise data extraction, condition filtering or calculation, and therefore, the profit data needs to be extracted precisely and the structured query formula information needs to be obtained. In order to improve the accuracy when dealing with precise requirements, at this time, the vector database also needs to be queried, and then the two databases are queried to further ensure the accuracy of the query result.

[0098] Specifically, the user query text needs to be vector encoded first to generate a query vector consistent with the vector dimension of the content block in the vector database; by calculating the similarity of the query vector and the atomic block, semantic block, and relationship block vector, the top N content blocks with the highest semantic correlation (such as "product AQ3 profit related row level semantic block" "profit calculation formula atomic block") are selected and integrated into the first data query result, i.e. the first data query result is to lock the semantic range related to the precise requirements, and then by combining the second data query result obtained by querying the relational database, the query result is more comprehensive and accurate.

[0099] Again, for the precise requirements in the query, a large language model (such as GPT-3.5 / 4) is called to convert the natural language description of the precise requirements (such as "extract product A profit data in Q3 of 2024" "filter products with unit price > 200 yuan and sales") into structured query statements (such as SQL) executable by the relational database; through the statement, the structured data of the matching conditions (such as "product A profit in Q3 of 2024: 5000 yuan" "products with unit price > 200 yuan: product B (sales 120 pieces), product C (sales 90 pieces)") is accurately extracted in the relational database as the second data query result.

[0100] Finally, the first data query result (semantically associated content block description, such as formula source, data association) and the second data query result (precise structured data) are integrated according to the logic of "core data first, then associated background". For example: first show the precise data: "Product A's profit in Q3 2024 is 5000 yuan (data source: table 1 row 5 column 5)"; then supplement the semantically associated content: "the corresponding profit calculation formula is atomic block-table 1 row 5 column 4 (LaTeX text: \text{profit}=\text{sales}\times(\text{unit price}-\text{cost}) ), which references table 1 row 5 column 2 (sales 100 pieces) and row 5 column 3 (unit price 80 yuan) data".

[0101] S2043, based on the query vector, query in the vector database, obtain the first data query result, and take the first data query result as the query result.

[0102] When the intent recognition result is that the query is only a fuzzy semantic requirement without precise data filtering / calculating requirement, for example, "find the formula and data source description related to cost accounting", only the associated content needs to be obtained through semantic similarity matching.

[0103] The first step of S2042 can be referred to for vector encoding of the user query text to generate a query vector, calculate the similarity in the vector database, and filter out the top N content blocks with the highest semantic association degree (such as "atomic block of cost accounting formula", "picture description block corresponding to sales trend chart", and "relationship block of profit calculation logic"), and integrate them into the first data query result.

[0104] Since there is no precise data requirement for the query, there is no need to trigger the large language model to generate a structured query statement and a relational database query process, and the first data query result is directly outputted in order of "similarity from high to low". For example: "1. Atomic block-table 1 row 3 column 4: cost accounting formula, LaTeX text: \text{cost}=\text{raw material cost}+\text{labor cost}; 2. Relationship block-table 1-002: this formula references table 1 row 3 column 2 (raw material cost) and row 3 column 3 (labor cost) atomic blocks; 3. Semantic block-table 2 row 6: product sales trend chart description-2024Q1-Q4 product A sales line chart, Q3 reaches the peak".

[0105] Through the subdivision of the query intent, a differentiated retrieval strategy is adopted to find the optimal solution between precision, efficiency, and result integrity, and finally output the query result that meets the user's real needs.

[0106] In the embodiments of the present application, for a query containing target requirements, the query is processed by combining the vector library semantic positioning and the relational database accurate extraction, so as to ensure the correlation between the results and the query semantics and the accuracy of the data; for a query not containing target requirements, only the vector library is used for retrieval, so as to simplify the process, reduce redundant calculation and realize resource on-demand allocation. Meanwhile, in the accuracy requirement scenario, the associated content of the vector library and the accurate data of the relational database are integrated, so as to avoid the problems of lack of background for searching accurate data and lack of accurate value for searching associated content in the traditional retrieval; in the fuzzy requirement scenario, the semantic matching is focused on, so as to ensure that the results are highly related to the query intention.

[0107] Further, the processes of obtaining the first data query result and the second data query result involved in S2042 and S2043 can refer to the following steps A-E:

[0108] Step A: converting the user query text into a vector to generate a corresponding query vector.

[0109] The user query text, i.e., the natural language text, is converted into a computer-calculable vector form, so that the query requirements can be compared with the content block vectors in the vector database at the semantic level.

[0110] Specifically, the same embedded model (for example, BERT or CLIP) as S203 (content block vector encoding) can be used to convert the user query text (for example, “Product A profit in Q3 of 2024 and calculation formula”) into a vector, and the output query vector has the same dimension. This ensures that the query vector and the content block vector are consistent in the semantic space, and the vectors generated based on the same model have reference value in the subsequent similarity calculation.

[0111] Step B: calculating the similarity scores between the query vector and the plurality of content block vectors stored in the vector database.

[0112] This step is to quantify the semantic correlation between the user query requirements and the content blocks in the vector database, and to find the content block that best meets the query intention.

[0113] Specifically, a vector similarity calculation algorithm (for example, cosine similarity or Euclidean distance) can be used to calculate the similarity scores between the query vector and all atomic block vectors, semantic block vectors and relational block vectors in the vector database (the score range is usually 0-1, and the closer to 1, the more similar the semantics).

[0114] For example, the similarity score between the query vector “Product A profit in Q3 of 2024” and the “row-level semantic block-table 1-row 5 (Product A Q3 data)” is 0.92, and the similarity score between the query vector and the “column-level semantic block-table 1-column 5 (all product profits)” is 0.85.

[0115] Step C, ranking the plurality of content blocks according to the similarity scores, selecting the top pre-set threshold number of content blocks with the highest similarity scores, and generating a first data query result based on the pre-set threshold number of content blocks.

[0116] Further, the most semantically relevant content is filtered from a large number of content blocks to form the first data query result. That is, all content blocks can be ranked from high to low according to the similarity scores; by setting a pre-set threshold (for example, the top 5), the top N content blocks with the highest scores are selected; the key information (such as block identifier, textual content, and association description) of these content blocks is extracted and integrated into the first data query result.

[0117] For example, when the pre-set threshold is 3, the final selection is "row-level semantic block-table 1-row 5", "atomic block-table 1-row 5-column 5 (profit data)", and "relationship block-table 1-001 (profit formula association description)", as the first data query result.

[0118] Step D, inputting the user query text into a large language model to generate a structured query statement corresponding to the user query text.

[0119] Specifically, this step converts the precise requirements of the user query text into a structured query statement (such as SQL) that can be directly executed by a relational database. The user query text can be input into a large language model with structured query statement conversion function, so that the model generates a corresponding structured query statement according to the precise requirements in the query (such as "extract the profit of product A in Q3 2024" and "query the cost accounting formula").

[0120] For example, when the user query text is "What is the profit of product A in Q3 2024", the SQL statement generated by the model can be represented as: SELECT target element content FROM cell table WHERE table ID = 1 AND row index = 5 AND column index = 5 AND target element content LIKE '% product A in Q3 2024 profit %'.

[0121] Step E, querying the relational database based on the structured query statement to obtain a second data query result.

[0122] Further, the structured query statement is used to accurately extract the quantitative data and specific content required by the user from the relational database to form the second data query result.

[0123] Specifically, the structured query statement generated in step D is submitted to the relational database, and the database returns the results that fully match the conditions after executing the statement as the second data query result.

[0124] For example, after executing the above SQL statement, the "target element content: product A profit in Q3 2024 - 5000 yuan" is returned as the second data query result.

[0125] The embodiment of the application converts the user query into a vector and calculates the similarity with the content block vector, which can break through the limitations of keyword matching, accurately identify semantically related content, and further filter the pre-set threshold number of content blocks to ensure that the first data query result focuses on core associated information and avoids irrelevant content interference.

[0126] At the same time, the natural language query of the user is converted into a structured query statement by a large language model, and based on the statement, the precise data meeting the conditions can be directly obtained in the relational database; and then the semantic associated background information (such as data source and associated relationship) is provided by the first data query result, and the second data query result provides accurate structured data, which improves the accuracy of the query result.

[0127] As an extension and refinement of the above embodiment, referring to Figure 4 The application also provides a construction process of a multi-level content block structure corresponding to a multi-modal table document in a multi-modal retrieval method, which can refer to the following steps:

[0128] S401, for each cell in any multi-modal table document, associate the target element content of the cell with the row and column indexes of the cell to generate a plurality of atomic blocks.

[0129] Specifically, for each cell of the multi-modal table, the target element content of each cell is associated and bound with the row index and column index of the cell to generate a plurality of independent atomic blocks.

[0130] Further, each cell information is given a spatial identifier to solve the problem of traditional table information only storing content without positioning, and then whether the single cell content is retrieved or the content source is traced, the corresponding atomic block can be quickly positioned through the row index and column index, ensuring the traceability of fine-grained information.

[0131] It should be noted that the refinement steps of S401 (for each cell in any multi-modal table document, associate the target element content of the cell with the row and column indexes of the cell to generate a plurality of atomic blocks) include the following steps a1-a3:

[0132] Step a1, traverse each cell in the multi-modal table document to obtain the target element content corresponding to each cell.

[0133] This step traverses each cell in the multi-modal table document one by one without missing any cell; for the multi-modal elements (text, pictures, formulas, etc.) in the cell, it is uniformly processed as standardized target element content. Avoid the defect of traditional table processing that only extracts text and ignores non-text elements, ensure that multi-modal information is completely collected, provide full and standardized content basis for atomic blocks, and eliminate the problem of ignoring non-text elements in subsequent retrieval.

[0134] Step a2, extract the row and column indexes corresponding to each cell, and associate the table identifier of the multi-modal table document corresponding to the cell.

[0135] For each traversed cell, extract its row index and column index in the table to determine its position in the table; at the same time, it is necessary to additionally associate the table identifier of the multi-modal table document to which the cell belongs. The table identifier is the unique identifier of each multi-modal table, which is used to distinguish tables in different documents.

[0136] In some embodiments, only the row index and column index cannot uniquely locate the cell (for example, "row 3 column 2" may correspond to different contents in "table ID_001" and "table ID_002"), and the present application assigns a globally unique position identifier to each cell by combining the table identifier, row index, and column index, solving the problem of repeated cell positions across tables and inability to accurately trace the source.

[0137] Step a3, bind the target element content, row index, column index, and table identifier of each cell to generate an atomic block.

[0138] The target element content, row index, column index, and table identifier of each cell are forcibly bound and packaged to form an independent atomic block.

[0139] For example, atomic block 1 = [table identifier: table ID_001; row index: row 3; column index: column 2; target element content: Markdown text "Product A, unit price 200 yuan"].

[0140] The present application ensures that all types of cell content such as text, pictures, and formulas can be converted into the target element content of the atomic block, avoiding the exclusion of non-text elements from the retrieval range, allowing subsequent retrieval to cover all information dimensions of the table. At the same time, by associating the index and the table identifier, each atomic block can be accurately located, and when the user queries the atomic block content, the position in the original table can be directly traced back; at the same time, this identifier also provides a key basis for subsequent row / column aggregation to generate semantic blocks.

[0141] S402, based on the row and column indexes of the cells, aggregating the atomic blocks corresponding to the cells by row, column, and the whole table to generate a plurality of semantic blocks corresponding to the row level, the column level, and the table level respectively.

[0142] Further, based on the atomic blocks generated in S401, the plurality of semantic blocks corresponding to the row level, the column level, and the table level are generated according to the row and column indexes of the cells.

[0143] It should be noted that the detailed steps of S402 (based on the row and column indexes of the cells, aggregating the atomic blocks corresponding to the cells by row, column, and the whole table to generate a plurality of semantic blocks corresponding to the row level, the column level, and the table level respectively) include the following steps b1-b3:

[0144] Step b1, based on the row index of the cell, aggregating the atomic blocks in the same row to generate a row-level semantic block.

[0145] Based on the row index of the cell, all atomic blocks (i.e. information of all cells in a row) in the same table with the same row index are packaged and integrated to form an independent row-level semantic block.

[0146] For example, if the atomic block of the table "row 5" contains "product B (column 1), 180 yuan per piece (column 2), 200 pieces (column 3), 36000 yuan (column 4)", the "row 5 row-level semantic block" after aggregation will uniformly carry the complete single-product business information of "product B's unit price 180 yuan per piece, sales 200 pieces, and revenue 36000 yuan".

[0147] Further, the scattered cell information is linked into a logically associated single row, meeting the user's demand for querying full-dimensional data of a certain subject (such as a certain product) (without the need to search each atomic block in the row one by one).

[0148] Step b2, based on the column index of the cell, aggregating the atomic blocks in the same column to generate a column-level semantic block.

[0149] Based on the column index of the cell, all atomic blocks (i.e. information of all cells in a column) in the same table with the same column index are packaged and integrated to form an independent column-level semantic block.

[0150] For example, if the atomic block of the table "column 3" (sales column) contains "150 pieces (row 4), 200 pieces (row 5), 120 pieces (row 6)", the "column 3 column-level semantic block" after aggregation will uniformly carry the summary information of the single column dimension "all product sales data: product A 150 pieces, product B 200 pieces, product C 120 pieces".

[0151] Integrate atomic blocks of the same dimension (such as all unit prices, all sales) into an ordered data sequence to meet the user's global summary demand for a certain dimension (such as all product unit prices) without traversing each row of the corresponding atomic block.

[0152] Step b3, integrate all row-level semantic blocks and column-level semantic blocks in the same table to generate a table-level semantic block.

[0153] This step takes table ownership as the aggregation basis to further integrate all row-level semantic blocks and all column-level semantic blocks within the same multi-modal table, extract the "core global information" of the table, and form a "table-level semantic block".

[0154] For example, a certain "2024Q4 product sales table" contains 6 row-level semantic blocks (corresponding to 6 products) and 4 column-level semantic blocks (product name, unit price, sales, and revenue). The aggregated "table-level semantic block" will carry the global overview of "2024Q4 product sales table: a total of 6 products, covering unit price 150-220 yuan / piece, sales 100-250 pieces, and revenue 15000-55000 yuan, with the core data dimensions of product name, unit price, sales, and revenue".

[0155] The embodiments of the present application integrate row and column semantics into a global correlation body through row-level semantic blocks, column-level semantic blocks, and table-level semantic blocks. This correlation allows the subsequent vector encoding to capture contextual semantics and improve semantic matching accuracy.

[0156] S403, based on the cross-modal correlation relationship, extracting the correlation relationship between atomic blocks and semantic blocks to generate a plurality of relationship blocks.

[0157] Based on the cross-modal correlation relationship parsed in the early stage, the correlation logic between atomic blocks and semantic blocks is extracted and described to generate relationship blocks. The correlation logic of multi-modal elements is established, and then during subsequent retrieval, the user can trace the original data block referenced by the formula, and associate the text block that explains the picture.

[0158] It should be noted that the detailed steps of the above S403 (based on the cross-modal correlation relationship, extracting the correlation relationship between atomic blocks and semantic blocks to generate a plurality of relationship blocks) include the following step c1:

[0159] Step c1, based on the cross-modal correlation relationship in the structure information, extracting the correlation relationship between different atomic blocks and the correlation relationship between different semantic blocks, and describing the correlation relationship in text form to construct a relationship block.

[0160] Based on the cross-modal association relationship in the previously parsed structural information, these associations are the implicit logic that naturally exists in multi-modal tables (such as "the text cell 'data source' points to the sales trend chart of the picture cell B4" "the formula cell C5 references the sales data cell A5 and the unit price data cell B5" "the row-level semantic block 'product A data' corresponds to the third data of the column-level semantic block 'profit column'").

[0161] Specifically, focusing on the cross-modal / cross-location association at the level of individual cells, such as "the text atomic block (row 2 column 1:'sales trend chart') and the picture atomic block (row 2 column 2: '2024Q1-Q4 line chart') are in a 'description and described' relationship" "the formula atomic block (row 5 column 4: 'profit = sales × (unit price - cost)') and the data atomic block (row 5 column 2: sales, row 5 column 3: unit price) are in a 'data reference' relationship".

[0162] And focusing on the association at the aggregation level, such as "the row-level semantic block (row 3: product A data) is subordinate to the column-level semantic block (column 5: profit column)" "the table-level semantic block (table 1: 2024Q3 sales data) contains the row-level semantic block (rows 3-8: 6 product data) and the column-level semantic block (columns 1-5: 5 indicator dimensions)".

[0163] Furthermore, the extracted association relationships are converted into standardized text (such as natural language descriptions or structured statements) to form independent "relationship blocks".

[0164] For example, a relationship block representing the relationship between atomic blocks can be represented as: [association type: data reference; association parties: formula atomic block (table 1-row 5-column 4) → data atomic block (table 1-row 5-column 2, table 1-row 5-column 3); corresponding text form description: the formula calculates profit by referencing sales and unit price data].

[0165] For another example, a relationship block representing the relationship between semantic blocks can be represented as: [association type: subordinate inclusion; association parties: row-level semantic block (table 1-row 3) → column-level semantic block (table 1-column 5); description: the profit data of product A is the third item of data in the profit column].

[0166] Furthermore, in traditional multi-modal tables, the correspondence between pictures and text, the reference between formulas and data, and other associations exist only in the table layout and are not explicitly recorded, so users cannot directly query which data a formula references or what the textual explanation of a picture is. The relationship blocks of the present application make these associations explicit through textual descriptions, and in subsequent retrieval, the system can locate the relevant relationship blocks through vector matching to achieve multi-dimensional information tracing.

[0167] S404, constructing a multi-level content block structure corresponding to the multi-modal table document according to the plurality of atomic blocks, the plurality of semantic blocks, and the plurality of relationship blocks.

[0168] Further, the atomic blocks generated in S401, the semantic blocks generated in S402, and the relationship blocks generated in S403 are integrated to form a three-layer structure of an atomic layer, a semantic layer, and a relationship layer, that is, a multi-level content block structure corresponding to the multi-modal table.

[0169] The three kinds of dispersed information units are converted into a hierarchical, logical, and linkable three-dimensional framework. The atomic layer is the foundation, the semantic layer is the aggregation and extension, and the relationship layer is the association link. The three together support the subsequent needs for granular retrieval and association tracing.

[0170] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0171] The embodiments of the present application also provide a multi-modal retrieval device, which corresponds to the method. Figure 5 A structural schematic diagram of a multi-modal retrieval device 500 provided by the present disclosure is provided. The device 500 of the present embodiment includes:

[0172] The receiving unit 51 is configured to receive a user query text.

[0173] The encoding unit 52 is configured to perform vector encoding on the user query text to obtain a query vector corresponding to the user query text.

[0174] The query unit 53 is configured to perform a query in a vector database according to the query vector to generate a query result. The vector database stores vector encodings corresponding to multi-level content block structures of a plurality of multi-modal table documents.

[0175] The multi-level content block structure of any of the multi-modal table documents includes an atomic layer composed of a plurality of atomic blocks, a semantic layer composed of a plurality of semantic blocks, and a relationship layer composed of a plurality of relationship blocks. The atomic block includes target element content and structure information of any cell in the multi-modal table document. The semantic block includes aggregated content corresponding to any row, or any column, or any entire table in the multi-modal table document. The relationship block includes text describing the association relationship between a plurality of atomic blocks and a plurality of semantic blocks. The structure information includes row and column indexes corresponding to each cell, and cross-modal association relationship between multi-modal elements.

[0176] As an optional implementation of the embodiment of the present application, the multi-modal retrieval device further comprises a construction unit, specifically configured to parse a plurality of multi-modal table documents, obtain target element content and structure information corresponding to each cell in each multi-modal table document, and construct a multi-level content block structure corresponding to the multi-modal table document based on the target element content and structure information corresponding to each cell for each multi-modal table document.

[0177] As an optional implementation of the embodiment of the present application, the query unit 53 is specifically configured to input the user query text into a target semantic recognition model, perform intent recognition on the semantics of the user query text through the target semantic recognition model, obtain an intent recognition result, query the vector database and the relational database according to the intent recognition result and the query vector, and generate the query result. The relational database stores structured content of a plurality of multi-modal table documents.

[0178] As an optional implementation of the embodiment of the present application, the query unit 53 is specifically configured to determine whether the intent recognition result contains a target requirement. The target requirement is any one of data aggregation, condition filtering, and multi-table association requirement. If yes, the query unit 53 performs a query in the vector database based on the query vector to obtain a first data query result, obtains a structured query statement corresponding to the user query text, performs a query in the relational database through the structured query statement to obtain a second data query result, integrates the first data query result and the second data query result, and generates the query result. If no, the query unit 53 performs a query in the vector database based on the query vector to obtain the first data query result, and takes the first data query result as the query result.

[0179] As an optional implementation of the embodiment of the present application, the query unit 53 is specifically configured to input the user query text into a large language model to generate a structured query statement corresponding to the user query text, and perform a query in the relational database based on the structured query statement to obtain the second data query result.

[0180] As an optional implementation of the embodiment of the present application, the query unit 53 is specifically configured to perform vector conversion on the user query text to generate a corresponding query vector, calculate a similarity score between the query vector and a plurality of content block vectors stored in the vector database, sort the plurality of content blocks according to the similarity score, select a preset threshold number of content blocks with the highest similarity, and generate the first data query result based on the preset threshold number of content blocks.

[0181] As an optional implementation of the embodiment of the present application, the construction unit is specifically configured to parse a plurality of multi-modal table documents, and extract initial element content in each cell in each of the multi-modal table documents. The initial element content includes at least one of a text element, a picture element, and a formula element. When the initial element content is the text element, the format of the text element is converted into a target lightweight markup language format to generate corresponding target element content. When the initial element content is the picture element, semantic understanding is performed on the picture element to generate a textual description corresponding to the picture element to generate corresponding target element content. When the initial element content is the formula element, the formula element is converted into a text in a target syntax format to generate corresponding target element content.

[0182] As an optional implementation of the embodiment of the present application, the construction unit is specifically configured to, for any one of the plurality of multi-modal table documents, extract a row and column index of each cell in the multi-modal table document in a multi-modal table document file, and a cross-modal association relationship between a text element, a picture element, and a formula element corresponding to the each cell to generate the structure information.

[0183] As an optional implementation of the embodiment of the present application, the construction unit is specifically configured to, for each cell in any of the multi-modal table documents, associate the target element content of the cell with the row and column index of the cell to generate a plurality of atomic blocks. The atomic blocks corresponding to the cell are aggregated according to the row and column index of the cell to generate a plurality of semantic blocks corresponding to rows, columns, and tables, respectively. The association relationship between the atomic blocks and the semantic blocks is extracted based on the cross-modal association relationship to generate a plurality of relationship blocks. The multi-level content block structure corresponding to the multi-modal table document is constructed according to the plurality of atomic blocks, the plurality of semantic blocks, and the plurality of relationship blocks.

[0184] The features of the embodiments of the multi-modal retrieval device can be referred to the related descriptions of the embodiments of the multi-modal retrieval method, which will not be repeated here.

[0185] Embodiments of the present application also provide an electronic device, comprising a memory and a processor, the memory storing a computer program, and the processor is configured to run the computer program to perform the steps in any of the above multi-modal retrieval method embodiments.

[0186] Embodiments of the present application also provide a computer readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above multi-modal retrieval method embodiments when running.

[0187] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0188] Embodiments of the present application also provide a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps in any of the above multi-modal retrieval method embodiments.

[0189] Embodiments of the present application also provide another computer program product comprising a non-volatile computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps in any of the above multi-modal retrieval method embodiments.

[0190] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0191] The above provides a detailed description of a multi-modal retrieval method and device. The principles and implementation methods of the present application are described by applying specific examples. The above example descriptions are only used to help understand the method and its core idea. It should be noted that for those skilled in the art, without departing from the principles of the present application, some improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A multimodal retrieval method, characterized in that, include: Receive user query text; The user query text is vector-encoded to obtain the query vector corresponding to the user query text; The query is performed in the vector database based on the query vector to generate query results; The vector database stores vector codes corresponding to the multi-level content block structures of multiple multimodal table documents; The multi-level content block structure of any of the multimodal table documents includes: an atomic layer composed of multiple atomic blocks, a semantic layer composed of multiple semantic blocks, and a relational layer composed of multiple relational blocks; the atomic blocks include the target element content and structural information of any cell in the multimodal table document; the semantic blocks include the aggregated content corresponding to any row, column, or entire table in the multimodal table document; the relational blocks include text describing the association relationships between multiple atomic blocks and multiple semantic blocks; the structural information includes the row and column indexes corresponding to each cell, as well as the cross-modal association relationships between multimodal elements; the semantic blocks are row-level, column-level, and table-level semantic blocks formed by aggregating the atomic blocks according to the row, column, and entire table dimensions based on the row and column indexes in the structural information. After performing vector encoding on the user query text to obtain the query vector corresponding to the user query text, the method further includes: The user query text is input into the target semantic recognition model, and the semantics of the user query text are used to perform intent recognition through the target semantic recognition model to obtain the intent recognition result; Based on the intent recognition result and the query vector, a query is performed using the vector database and the relational database to generate the query result; the relational database stores the structured content of multiple multimodal table documents; The step of generating the query result by querying the vector database and the relational database based on the intent recognition result and the query vector includes: Determine whether the intent recognition result contains the target requirement; the target requirement is any precise data query requirement among data aggregation, conditional filtering, and multi-table join requirements; If so, then based on the query vector, a query is performed in the vector database to obtain a first data query result, and a structured query statement corresponding to the user query text is obtained. The structured query statement is then used to perform a query in the relational database to obtain a second data query result. The first data query result and the second data query result are then integrated to generate the query result. If not, then a query is performed on the vector database based on the query vector to obtain the first data query result, and the first data query result is used as the query result.

2. The method according to claim 1, characterized in that, Before receiving the user's query text, the method further includes: Parse multiple multimodal table documents to obtain the target element content and structural information corresponding to each cell in each multimodal table document; For each of the multimodal table documents, a multi-level content block structure is constructed based on the target element content and structural information corresponding to each cell. The multi-level content blocks are vector-encoded and stored in the vector database.

3. The method according to claim 1, characterized in that, The step of querying the vector database based on the query vector to obtain the first data query result includes: The user query text is vectorized to generate a corresponding query vector; Calculate the similarity score between the query vector and multiple content block vectors stored in the vector database; The multiple content blocks are sorted according to the similarity score, and the top preset threshold of the content blocks with the highest similarity are selected. The first data query result is generated based on the preset threshold of the content blocks.

4. The method according to claim 1, characterized in that, The step of querying the relational database using the structured query statement to obtain the second data query result includes: The user query text is input into the large language model to generate a structured query statement corresponding to the user query text; Based on the structured query statement, a query is performed in the relational database to obtain the second data query result.

5. The method according to claim 2, characterized in that, The parsing of multiple multimodal table documents includes: The process involves parsing multiple multimodal table documents and extracting the initial element content from each cell in each multimodal table document; the initial element content includes at least one of text elements, image elements, and formula elements. When the initial element content is the text element, the format of the text element is converted into the target lightweight markup language format to generate the corresponding target element content; When the initial element content is the image element, semantic understanding is performed on the image element to generate a text description corresponding to the image element, so as to generate the corresponding target element content; When the initial element content is the formula element, the formula element is converted into text in the target syntax format to generate the corresponding target element content.

6. The method according to claim 2, characterized in that, The parsing of multiple multimodal table documents includes: For any one of the multiple multimodal table documents, extract the row and column indices of each cell in the multimodal table document file, as well as the cross-modal relationships between the text elements, image elements, and formula elements corresponding to each cell, to generate the structural information.

7. The method according to claim 2, characterized in that, For each of the multimodal table documents, based on the target element content and structural information corresponding to each cell, a multi-level content block structure is constructed for the multimodal table document, including: For each cell in any of the multimodal table documents, the content of the target element in the cell is associated with the row and column index of the cell to generate multiple atomic blocks; Based on the row and column indexes of the cell, the atomic blocks corresponding to the cell are aggregated by row, column and whole table to generate multiple semantic blocks corresponding to row level, column level and table level respectively; Based on the cross-modal association relationship, the association relationships between the atomic blocks and between the semantic blocks are extracted to generate multiple relation blocks; Based on the multiple atomic blocks, multiple semantic blocks, and multiple relational blocks, a multi-level content block structure corresponding to the multimodal table document is constructed.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the multimodal retrieval method as described in any one of claims 1-7.

Images