Visual information retrieval question-answering system and method based on multi-modal embedding and visual language model

Abstract

The invention discloses a visual information retrieval question-answering system and method based on multi-modal embedding and a visual language model, and relates to the technical field of information retrieval and natural language processing. The system comprises an off-line indexing construction module and an on-line retrieval question and answer module, user text questions and image data in a database are uniformly mapped to the same high-dimensional vector space through a multi-modal embedding model, efficient and accurate vector similarity retrieval is achieved, then retrieved related images and original questions are input into a visual big language model, and the visual big language model is built. And an accurate answer is directly generated. According to the method, a traditional image-to-text conversion link is skipped, visual context information is completely reserved, the question and answer retrieval accuracy and efficiency are improved, multi-modal data processing and privacy deployment are supported, and the method is suitable for multiple professional fields such as finance, medical treatment and commercial analysis.

Description

Technical Field

[0001] This invention relates to the fields of information retrieval and natural language processing, specifically to a visual information retrieval question-answering system and method based on multimodal embedding and visual language models. Background Technology

[0002] In various scenarios such as project management, business analysis, financial services, and medical diagnosis, a wealth of valuable information is encapsulated within visual elements such as images, charts, tables, and complex documents (e.g., PDF reports, PowerPoint presentations). This unstructured visual information carries key data, logical relationships, and decision-making basis, and its efficient and accurate retrieval and interpretation are crucial for business operations.

[0003] Traditional computer systems typically rely on Optical Character Recognition (OCR) or complex image-to-text / Markdown conversion techniques when processing this unstructured visual information. However, these methods have many problems.

[0004] First, there is a significant loss of information. OCR technology struggles to accurately recognize complex tables, charts, and handwriting. Furthermore, the conversion process completely loses rich visual contextual information such as the layout, color, arrow direction, and element relationships of the image, making it impossible for subsequent processing to obtain complete information dimensions. For example, it is impossible to accurately determine the height relationship of each bar in a bar chart or the logical direction of arrows in a flowchart.

[0005] Secondly, the existing technology has a cumbersome and inefficient processing flow. Preprocessing, converting and cleaning massive amounts of images is a computationally intensive and time-consuming process. It not only requires a lot of computing resources, but also makes it difficult to achieve real-time information retrieval and response. When the image data changes, the entire process needs to be reprocessed, which is extremely costly and inefficient.

[0006] Moreover, existing technologies have low retrieval and question-answering accuracy. Information loss and errors in the front-end conversion steps can prevent subsequent language models from obtaining complete and accurate context, resulting in the final retrieval results deviating from the requirements, and the question-answering results being incorrect or not the optimal solution. This fails to meet the high requirements for information accuracy in professional fields such as finance, healthcare, and consulting.

[0007] Therefore, the industry urgently needs a technical solution that can directly understand and process complex image content, avoid information distortion during the conversion process, and thus achieve more efficient and accurate multimodal information retrieval and question answering. Summary of the Invention

[0008] In summary, in response to the shortcomings of existing technologies, this invention proposes a visual information retrieval question-answering system and method based on multimodal embedding and visual language models. It aims to solve the problems of image information conversion loss, low processing efficiency, and insufficient accuracy of retrieval and question answering in traditional technologies, and achieve efficient and accurate retrieval and question answering of visual information.

[0009] Specifically, this invention provides a visual information retrieval question-answering method based on multimodal embedding and visual language models. The method includes an offline index construction stage and an online retrieval question-answering stage. The offline index construction stage includes image traversal, image vectorization, and index library construction steps. The online retrieval question-answering stage includes question input, question vectorization, vector similarity retrieval, multimodal prompt construction, model inference, and answer generation steps. In the offline index construction stage, the image traversal step traverses all images in the image document database; the image vectorization step calls the multimodal embedding model to perform feature extraction and vector transformation on each traversed image, generating high-dimensional image vectors that can represent the core content of the image; the index library construction step stores all high-dimensional image vectors in a vector database to construct a vector index library. In the online question-and-answer retrieval phase, the question input step allows users to input text questions through the system's interactive interface; the question vectorization step calls the same multimodal embedding model as the offline index building phase to transform the text questions into high-dimensional query vectors; the vector similarity retrieval step, based on the high-dimensional query vectors, performs high-speed similarity calculations in a pre-built vector index library, calculating the similarity value between the query vector and each image vector, sorting them from high to low similarity values, and selecting the top N most similar images as target images; the multimodal prompt construction step combines the user's original text questions with the N retrieved target images to generate multimodal input prompts; the model inference step sends the multimodal input prompts to the visual big language model, which parses the intent of the text questions and simultaneously performs deep understanding of the target images, combining text and image information for comprehensive inference; the answer generation step allows the visual big language model to generate the final answer in natural language format based on the results of comprehensive inference.

[0010] Preferably, the multimodal embedding model has cross-modal feature mapping capability to ensure that the image vector can accurately reflect visual information.

[0011] Preferably, in the index database construction step, when image data is added, deleted, or modified in the image document database, the corresponding image vectors and vector index database are updated synchronously.

[0012] Preferably, the text question includes data querying, logical judgment, and relationship recognition in the image.

[0013] Preferably, in the question vectorization step, the query vector and the image vector are in the same high-dimensional vector space.

[0014] Preferably, in the vector similarity retrieval step, the positive integer N is set according to actual needs.

[0015] Optionally, N=3.

[0016] Preferably, in the model reasoning step, the visual big language model performs in-depth understanding of the chart data, layout structure, and logical relationships in the target image, and identifies the meaning of the rows and columns of the table in the image, extracts the specific values ​​in the bar chart, and sorts out the logical links in the flowchart.

[0017] Preferably, the multimodal embedding model includes an image encoder responsible for image vectorization and a text encoder responsible for text question vectorization. The multimodal embedding model can project the outputs of the image encoder and the text encoder to a feature space of a unified dimension through contrastive learning.

[0018] The present invention also provides a visual information retrieval question-answering system based on multimodal embedding and visual language model, for performing the methods described above.

[0019] The visual information retrieval question-answering system and method based on multimodal embedding and visual language models provided by this invention maps user text questions and image data in the database to the same high-dimensional vector space through a multimodal embedding model, achieving efficient and accurate vector similarity retrieval. The retrieved relevant images and the original question are then input into a visual large-scale language model to directly generate an accurate answer. This invention skips the traditional image-to-text conversion step, fully preserving visual contextual information, improving the accuracy and efficiency of retrieval and question answering, supporting multimodal data processing and privacy deployment, and is applicable to multiple professional fields such as finance, healthcare, and business analytics. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be discussed below. Obviously, the technical solutions described in conjunction with the accompanying drawings are only some embodiments of the present invention. For those skilled in the art, other embodiments and their accompanying drawings can be obtained based on the embodiments shown in these drawings without creative effort.

[0021] Figure 1 A basic architecture diagram of a visual information retrieval question-answering system based on multimodal embedding and visual language model according to the present invention is shown. Figure 2 The flowchart of the visual information retrieval question answering method based on multimodal embedding and visual language model is shown. Detailed Implementation

[0022] The technical solutions of various embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments described in the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] In summary, in response to the shortcomings of existing technologies, this invention proposes a visual information retrieval question-answering system and method based on multimodal embedding and visual language models. It aims to solve the problems of image information conversion loss, low processing efficiency, and insufficient accuracy of retrieval and question answering in traditional technologies, and achieve efficient and accurate retrieval and question answering of visual information.

[0024] To achieve the above objectives, the core concept of this invention is to realize cross-modal unified representation of textual questions and image data through a multimodal embedding model, quickly retrieve relevant images based on vector similarity, and then directly analyze the images and questions using a visual large language model to generate accurate answers, skipping the traditional image-to-text conversion process and fundamentally solving the problem of information loss.

[0025] The specific architecture and operation flow of the method and system provided by the present invention will be described in detail below with reference to the accompanying drawings. Figure 1 The basic architecture diagram of the visual information retrieval question-answering system based on multimodal embedding and visual language model according to the present invention is shown. Figure 2 The flowchart of the visual information retrieval question answering method based on multimodal embedding and visual language model is shown.

[0026] The system described in this invention includes an offline index building module and an online retrieval and question answering module. The specific functions of each module will be described in detail below.

[0027] First, we will introduce the offline index building module. This module is used to preprocess image data in the image / document database, generate image vectors, and build a vector index library, providing a foundation for subsequent online retrieval. This module includes an image traversal unit, a multimodal embedding unit, and an index building unit. These three units will be described below.

[0028] The image traversal unit can traverse all image data in the image / document database, including but not limited to PPT pages, PDF pages, chart screenshots, table images, handwritten images, and other carriers containing visual information.

[0029] The multimodal embedding unit can call a pre-defined multimodal embedding model, which supports cross-modal feature mapping between text and images. This model can transform the visual features of an image into a high-dimensional vector, generating an image vector representing the core content of each image. This multimodal embedding model can project the outputs of the image encoder responsible for image vectorization and the text encoder responsible for text question vectorization to a feature space of the same dimension through contrastive learning.

[0030] The index building unit stores all generated image vectors in a dedicated vector database to construct a vector index library that supports efficient storage and retrieval of high-dimensional vectors. The vector database has high-speed similarity calculation capabilities, providing support for subsequent rapid retrieval.

[0031] The online search and question-and-answer module will then be introduced.

[0032] The online question-and-answer module receives user text questions, performs retrieval and question-and-answer based on the vector index library generated by the offline index building module, generates the final answer, and returns it to the user. This module includes a question vectorization unit, a vector retrieval unit, a multimodal prompt building unit, a visual language model inference unit, and an answer generation unit. These units will be described in detail below.

[0033] The question vectorization unit receives text questions input by users online, calls the same multimodal embedding model as in the offline index building stage, and transforms the semantic features of the text questions into high-dimensional query vectors, ensuring that the questions and images are similar to each other in the same vector space.

[0034] The vector retrieval unit then performs high-speed similarity calculations (e.g., using a cosine similarity algorithm) in the vector index library based on the generated query vector, and filters out at least one target image that is most relevant to the query vector. The filtering process can be achieved by setting a similarity threshold or by selecting the top N (N is a positive integer) most similar images.

[0035] The multimodal prompt building unit combines the user's original text question with the retrieved target image to generate a multimodal input prompt that meets the input requirements of the visual large language model, ensuring that the model can simultaneously obtain textual intent and image information.

[0036] The visual language model reasoning unit inputs multimodal input prompts into a preset visual language model. This visual language model has powerful image understanding and logical reasoning capabilities, such as Qwen2.5-VL from Tongyi Qianwen and Doubao-1.5-vision-pro from Doubao. It can "understand" the chart data, layout relationships, and logical connections in images like a human, and perform deep reasoning in combination with text questions.

[0037] The answer generation unit receives the reasoning results from the visual large language model, converts them into the final answer in natural language format, and returns it to the user, thus achieving accurate question answering based on visual information.

[0038] Furthermore, the system architecture of this invention supports deployment in a virtual private cloud (VPC) or local environment, meeting the stringent data security and privacy requirements of regulated industries such as finance and healthcare; it also supports processing various data types such as text, images, complex charts, and tables, and is compatible with multilingual input, making it suitable for global business scenarios.

[0039] The following section will further describe the relevant method flow of the present invention based on the above system architecture.

[0040] The method described in this invention includes two stages: offline index construction and online retrieval and question answering. The specific process is described below.

[0041] The first stage is the offline index building stage, also known as the preprocessing stage. The steps in this stage are described one by one below.

[0042] Image traversal steps: Traverse all image and document data in the image / document database. The image data includes various visual information carriers such as PPT pages, PDF pages, chart screenshots, table images, and handwritten images to ensure that all image documents to be retrieved are processed without omission.

[0043] Image vectorization steps: Call the multimodal embedding model to extract features and transform vectors for each traversed image, generating a high-dimensional image vector that can represent the core content of the image. The multimodal embedding model must have cross-modal feature mapping capability to ensure that the image vector can accurately reflect visual information.

[0044] Index database construction steps: Store all generated image vectors in a vector database to build a vector index database. This vector database must support efficient storage of high-dimensional vectors and fast similarity retrieval, providing a foundation for subsequent online retrieval. This stage is independent of the online retrieval and question-answering stage and can be executed periodically in the background. When image data is added, deleted, or modified in the image / document database, the corresponding image vectors and vector index database are updated synchronously to ensure the timeliness of the index database.

[0045] The synchronous update here fully embodies the dynamic update feature of this invention. Dynamic index updates mean that when image data in the image document database is added, deleted, or modified, the system does not need to fully reconstruct the entire vector index database. Instead, it achieves real-time or near-real-time synchronous updates of the vector index database by accurately locating the changed data and efficiently processing related vectors. This dynamic update comprehensively considers ensuring the consistency between the index database and the original image database, ensuring the accuracy of search results, minimizing computational resource consumption during the update process, avoiding the inefficiency caused by full reconstruction, and maintaining the continuity of the system's online search service, with no interruption or minimal impact on user queries during the update.

[0046] For example, when adding new image data, the new images are filtered by the metadata of the image database (such as file creation time and unique identifier ID) to generate a list of new images, ensuring that there is no duplicate processing.

[0047] When deleting image data, the system queries the "Image Vector - Original Image Data" association table based on the unique identifier of the deleted image to accurately locate the high-dimensional vector to be deleted and avoid accidentally deleting other vectors.

[0048] When modifying image data, the system uses the unique identifier of the modified image data to simultaneously locate the old vector corresponding to the original version and the newly added vector of the modified image, thereby completing the modification and update without affecting other image data that does not need to be modified.

[0049] The process then proceeds to the online search and question-and-answer stage, i.e., the real-time processing stage. The steps in this stage are described below.

[0050] Question input steps: Users input text questions through the system's interactive interface. These questions can involve various needs such as data querying in images, logical judgment, and relationship recognition, for example, "What is the revenue data of a certain company in Q3 2024?" "What is the core link after step A in the flowchart?" etc.

[0051] Question vectorization steps: The system calls the same multimodal embedding model as in the offline index building stage to convert the user-input text question into a high-dimensional query vector, ensuring that the query vector and the image vector are in the same high-dimensional vector space, thus guaranteeing the accuracy of similarity calculation.

[0052] Vector similarity retrieval steps: Based on the query vector, the system performs high-speed similarity calculation in a pre-built vector index library (e.g., using the cosine similarity algorithm) to calculate the similarity value between the query vector and each image vector. After sorting the similarity values ​​from high to low, the system selects the top N (N is a positive integer, which can be set according to actual needs, such as N=3) most similar images as the target images.

[0053] Multimodal prompt construction steps: The system combines the user's original text question with the retrieved N target images to generate multimodal input prompts, such as "Based on the following images, answer the question: What is the revenue data of a certain company in Q3 2024? [Image 1][Image 2][Image 3]", ensuring that the visual big language model can simultaneously obtain textual intent and relevant image information.

[0054] Model inference steps: The multimodal input prompts are sent to the backend visual big language model. This model parses the intent of the text question and performs a deep understanding of the chart data, layout structure, and logical relationships in the target image. It combines text and image information to perform comprehensive reasoning, such as recognizing the meaning of rows and columns in a table in an image, extracting specific values ​​from a bar chart, and sorting out the logical links in a flowchart.

[0055] Answer generation steps: Based on the reasoning results, the visual big language model generates the final answer in natural language format, such as "According to the financial statement in the image, the revenue of a certain company in Q3 2024 was 1.25 billion yuan, an increase of 8.3% year-on-year", and returns it to the user through the system interface to complete the search and answer process.

[0056] To illustrate the technical solution of the present invention more clearly and intuitively, the following detailed description is provided in conjunction with specific embodiments.

[0057] This embodiment is set as a financial report retrieval question and answer scenario in the financial industry. In this scenario, a commercial bank needs to conduct a retrieval question and answer on a massive amount of corporate financial reports (stored in PDF format, containing a large number of financial tables, revenue bar charts, profit line charts and other visual elements) to quickly obtain the financial data of the target company for credit assessment and risk analysis. It is required to ensure data accuracy and retrieval efficiency, while meeting the data privacy requirements of the financial industry.

[0058] The corresponding system deployment and configuration include, for example, deploying the system described in this invention in a Virtual Private Cloud (VPC) to ensure that data is not leaked externally and to meet the privacy compliance requirements of the financial industry. Specifically, the visual information retrieval question-answering system is deployed in a private cloud environment or a local data center, and the image vectorization and model inference processes are completed in a physically isolated internal network. The multimodal embedding model uses the CLIP model (Contrastive Language-Image Pretraining), which supports cross-modal text-image mapping. The visual large language model can be the Doubao model, which has powerful table recognition and data extraction capabilities. The vector database can be the Milvus vector database, which supports efficient storage and millisecond-level retrieval of high-dimensional vectors. The image / document database stores 100,000 companies' financial report PDF files from 2019 to 2024, and the system automatically converts each page of the PDF into image format for processing.

[0059] Under the above scenario and system deployment, the following steps should be implemented: Image traversal steps: The system traverses all financial report PDF files in the image / document database, converting each PDF file into a PNG image page by page, generating a total of 1.2 million images (an average of 12 pages of financial report per company).

[0060] Image vectorization steps: Call the CLIP model to extract features from each financial report image and generate a 768-dimensional image vector. For example, for a company's revenue bar chart for Q2 2024, the CLIP model extracts its core features such as "2024Q2", "revenue", "bar chart", and "value range of 500-800 million yuan" and converts them into corresponding high-dimensional vectors. Index building steps: Store the vectors of 1.2 million images in the Milvus vector database to build a vector index. Set the index type to IVF_FLAT and the number of cluster centers to 1024 to ensure efficient similarity retrieval. Simultaneously, establish a mapping between image vectors and the original financial report PDFs for subsequent source tracing.

[0061] Next, we will conduct an online search and answer process. The specific steps are as follows: Question input steps: The bank credit manager inputs the text question through the system interface: "What was the net profit of Technology Company A in the first half of 2024? What was the year-on-year growth rate?"

[0062] Question vectorization steps: The system calls the CLIP model to convert the text question into a 768-dimensional query vector, which represents core semantics such as "Company A", "first half of 2024", "net profit", and "year-on-year growth rate".

[0063] Vector similarity retrieval steps: The system calculates cosine similarity based on the query vector in the Milvus vector database and selects the three images with the highest similarity, namely the "Profit Statement" page image, the "2024 First Half-Year Net Profit Year-on-Year Analysis" bar chart image, and the "Summary of Financial Core Indicators" table image from the PDF of Company A's 2024 semi-annual report.

[0064] Multimodal prompt construction steps: The system generates multimodal input prompts: "Based on the following images, answer the question: What was the net profit of Technology Company A in the first half of 2024? What was the year-on-year growth rate? [Image 1: Profit and Loss Statement] [Image 2: Year-on-Year Net Profit Bar Chart] [Image 3: Summary of Core Indicators]"; Model inference steps: Input the multimodal input prompt into the Doubao-1.5-vision-pro model. The model first parses the question to determine the intent of the query, which is to query the net profit amount and year-on-year growth rate of a specific company in a specific period. Then, it analyzes Image 1 (profit statement) and identifies the column "First Half of 2024" with a value of 320 million yuan for "Net Profit". It analyzes Image 2 (bar chart) and extracts the net profit for the first half of 2023 as 260 million yuan, calculating the year-on-year growth rate as (320 million - 260 million) / 260 million ≈ 23.08%. It analyzes Image 3 (core indicator summary) to verify that the net profit and year-on-year growth rate data are consistent, ensuring the accuracy of the results.

[0065] Answer generation steps: The model generates the final answer: "According to the relevant financial data in the 2024 semi-annual report of Company A, the company's net profit in the first half of 2024 was 320 million yuan, and the net profit in the first half of 2023 was 260 million yuan, with a year-on-year growth rate of approximately 23.08%." and returns it to the credit manager.

[0066] The coordinated execution of these steps enables direct analysis of the original financial statement image, avoiding table formatting errors and numerical inconsistencies caused by OCR conversion. Net profit and year-on-year growth rate data are completely consistent with the original financial statement, significantly improving accuracy. Efficiency is also greatly enhanced; the time from user input to obtaining the answer is only a few seconds, a stark contrast to the traditional method of manually flipping through PDFs and extracting data (averaging 30 minutes). Furthermore, the process ensures privacy and security; the VPC deployment mode guarantees that financial statement data is not leaked, meeting the data privacy compliance requirements of the financial industry.

[0067] This concludes the basic introduction of the present invention. As mentioned above, the present invention effectively overcomes many defects in the prior art and achieves many significant beneficial effects compared to the prior art.

[0068] First, the system and method provided by this invention can achieve ultimate accuracy, skipping the "image-to-text" conversion process in traditional technologies where information loss is severe, and directly processing and understanding the original image. It fully preserves all visual contextual information such as the image's layout, color, and logical relationships, fundamentally avoiding information distortion during the conversion process, greatly improving the accuracy of retrieval and question answering, and meeting the high requirements of professional fields.

[0069] It can also achieve unprecedented efficiency, eliminating the need for complex image preprocessing, conversion and cleaning processes. It enables second-level retrieval of massive images through multimodal embedding models and vector databases. Combined with the real-time reasoning capabilities of the visual big language model, it enables rapid return of question-and-answer results, greatly saving manpower and time costs.

[0070] Furthermore, it can achieve powerful multimodal processing capabilities, supporting the processing of various data types such as text, images, complex charts, tables, and handwriting, and is compatible with multilingual input. It is suitable for multiple fields such as project management, business analysis, finance, and healthcare, meeting the needs of global business scenarios.

[0071] In addition, this invention provides comprehensive privacy and security protection, supports virtual private cloud (VPC) or local deployment mode, and all data processing is completed in a private environment without being disclosed to the outside world, meeting the strict requirements of regulated industries such as finance and healthcare for data security and privacy.

[0072] Meanwhile, the system of the present invention also has flexible scalability. The vector index library supports dynamic updates. When image data in the image / document database is added, deleted or modified, the corresponding image vectors and indexes can be updated synchronously to ensure that the system can adapt to constantly changing data sources and has good scalability and practicality.

[0073] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A visual information retrieval and question-answering method based on multimodal embedding and visual language models, characterized in that, This method includes an offline index building stage and an online retrieval and question answering stage. The offline index building stage includes image traversal, image vectorization, and index database construction steps. The online retrieval and question answering stage includes question input, question vectorization, vector similarity retrieval, multimodal prompt construction, model inference, and answer generation steps. In the offline index building phase, the image traversal step traverses all images in the image document database; the image vectorization step calls the multimodal embedding model to perform feature extraction and vector transformation on each traversed image, generating high-dimensional image vectors that can represent the core content of the image; the index library building step stores all high-dimensional image vectors into the vector database to build the vector index library. In the online question-and-answer retrieval phase, the question input step allows users to input text questions through the system's interactive interface; the question vectorization step calls the same multimodal embedding model as the offline index building phase to transform the text questions into high-dimensional query vectors; the vector similarity retrieval step performs high-speed similarity calculations based on the high-dimensional query vectors in a pre-built vector index library, calculates the similarity value between the query vector and each image vector, sorts them from high to low similarity value, and selects the top N most similar images as target images; The multimodal prompt construction step combines the user's original input text question with N retrieved target images to generate multimodal input prompts; The model inference step sends multimodal input prompts to the visual big language model, which parses the intent of the text question and performs a deep understanding of the target image, combining text and image information for comprehensive reasoning; the answer generation step allows the visual big language model to generate the final answer in natural language format based on the results of comprehensive reasoning.

2. The method according to claim 1, characterized in that, The multimodal embedding model has cross-modal feature mapping capabilities, ensuring that image vectors can accurately reflect visual information.

3. The method according to claim 1, characterized in that, During the index building process, when image data is added, deleted, or modified in the image document database, the corresponding image vectors and vector index are updated synchronously.

4. The method according to claim 1, characterized in that, The text questions include data queries, logical judgments, and relationship recognition in the image.

5. The method according to claim 1, characterized in that, In the question vectorization step, the query vector and the image vector reside in the same high-dimensional vector space.

6. The method according to claim 1, characterized in that, In the vector similarity retrieval step, N is a positive integer between 1 and 10, and the positive integer N is set according to actual needs.

7. The method according to claim 1, characterized in that, In the model inference step, the visual big language model performs a deep understanding of the chart data, layout structure, and logical relationships in the target image, and identifies the meaning of the rows and columns in the table in the image, extracts the specific values ​​in the bar chart, and sorts out the logical links in the flowchart.

8. The method according to claim 1, wherein the multimodal embedding model includes an image encoder responsible for image vectorization and a text encoder responsible for text question vectorization, and the multimodal embedding model is capable of projecting the outputs of the image encoder and the text encoder to a feature space of a unified dimension through contrastive learning.

9. A visual information retrieval question-answering system based on multimodal embedding and visual language model, for performing the method according to any one of claims 1-8.

10. The system according to claim 9, wherein the visual information retrieval question answering system is deployed in a private cloud environment or local data center, and the image vectorization and model inference processes are both completed in a physically isolated internal network.

Images