Deep understanding and heterogeneous fusion of image, text, audio and video cross-modal mutual inspection method and system

Abstract

The application relates to the technical field of information retrieval, in particular to a picture-text-audio-video cross-modal mutual retrieval method and system based on deep document understanding and heterogeneous data fusion, and the method comprises the following steps: 1. data access and preprocessing, 2. deep analysis and conversion of each modality text, 3. deep analysis and conversion of each modality text, 4. construction of a "keyword-vector" double index system, and 5. cross-modal retrieval. The application solves the problems of semantic gap, insufficient document analysis precision and insufficient heterogeneous data fusion in the prior art cross-modal retrieval, improves retrieval accuracy and relevance, enhances system user friendliness and customizability, supports one-stop processing and retrieval of various heterogeneous data sources, and efficiently realizes the function of retrieving other modality data of a knowledge base from any one of picture-text-audio-video modality data.

Description

Technical Field

[0001] This invention relates to the field of information retrieval technology, specifically to a cross-modal mutual inspection method and system for images, text, audio, and video that combines deep understanding and heterogeneous fusion. Background Technology

[0002] With the rapid development of multimedia technology, heterogeneous data such as text, documents, images, audio, and video have become the main carriers of information dissemination, and cross-modal retrieval has become a core requirement in scenarios such as knowledge management and content recommendation; for example, retrieving information related to other modalities through data from any modality, such as using product images to retrieve instruction manual text, or using meeting recording clips to match corresponding demonstration videos.

[0003] Existing cross-modal retrieval technologies mainly achieve this through the following technical methods: Single-modal retrieval extension: Based on traditional text retrieval, OCR technology is used to convert text in images into text, or speech recognition is used to convert audio into text, and then cross-modal matching is achieved based on text retrieval logic.

[0004] Multimodal feature space construction and combination: First, use deep learning models to build independent feature indexes for text, images, audio and video, such as text keyword indexes and image visual feature indexes. Then, use contrastive learning and other techniques to map different modal features to the same space, and realize retrieval through vector similarity calculation.

[0005] Simple data fusion: Structured data such as Excel and unstructured data such as PDF are indexed independently, and the results are matched separately and then merged during retrieval, without achieving deep semantic association.

[0006] Existing technologies suffer from the following key problems, resulting in low accuracy and limited applicability in cross-modal retrieval: Significant cross-modal semantic gap exists, and the fusion degree of heterogeneous data features is low.

[0007] On the one hand, existing technologies have not constructed a unified semantic representation space. The underlying single-modal retrieval extension mode is based on text retrieval, which cannot handle complex semantic relationships. Multimodal data processing is fragmented, with different modalities operating independently: features of text, image, and audio modalities are generated based on their own independent models, such as text relying on language models and images relying on visual models. The feature dimensions and semantic mapping logic differ greatly. During retrieval, only single-modal results are simply concatenated, resulting in the inability to match information that is "semantically equivalent but feature-heterogeneous." The lack of deep fusion of cross-modal semantic associations leads to poor relevance of results. On the other hand, the cost barrier for multimodal semantic unification is extremely high. In existing technologies, to achieve semantic unification of multimodal data such as text, image, audio, and video through a single model, i.e., to make "pictures of cats" and "textual descriptions of 'cats'" semantically equivalent, it is necessary to rely on ultra-large-scale training data, usually requiring billions of cross-modal aligned samples and huge computational investment. Most of them use models with hundreds of billions of parameters, making it difficult for small and medium-sized institutions to afford. Moreover, data annotation needs to take into account the semantic associations between modalities, further raising the threshold for technology implementation.

[0008] The parsing accuracy of complex documents is insufficient, and the depth of understanding is inadequate.

[0009] Existing technologies are weak in processing documents containing structured information, such as financial reports with tables and scientific research papers with multi-region layouts: they only extract text through OCR, ignoring structural information such as row and column relationships and merged cells in tables. This causes key data in the document, such as the "correlation between revenue values ​​and corresponding years" in tables, to be broken down into isolated text, which cannot participate in semantic retrieval and directly affects the retrieval accuracy of document-type data.

[0010] Inflexible search strategies.

[0011] At the retrieval strategy level, most systems adopt a fixed "one-size-fits-all" logic: for example, regardless of whether the user inputs text, image or audio, the same set of retrieval weights is applied, such as keyword matching accounting for 70% and semantic vector matching accounting for 30%. This cannot respond to the characteristics of different modalities, such as text relying more on precise keywords and images relying more on semantic associations, nor can it adapt to diverse scenario needs.

[0012] Among the aforementioned technical problems, the cross-modal semantic gap and insufficient parsing accuracy of complex documents are among the core obstacles restricting the development of existing technologies: the cross-modal semantic gap prevents deep semantic association between data of different modalities, and insufficient parsing accuracy of complex documents leads to the loss of key information in document-type data. Both of these factors contribute to the phenomenon of "low recall and poor relevance" in cross-modal retrieval. In addition, rigid retrieval strategies further amplify these defects, making it difficult for existing systems to meet the high-precision retrieval needs of real-world scenarios. Summary of the Invention

[0013] The purpose of this invention is to provide a method and system for cross-modal mutual inspection of images, text, audio and video that combines deep understanding and heterogeneous fusion, so as to achieve accurate association and efficient retrieval of multimodal data.

[0014] The present invention provides a cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion, comprising the following steps: Step 1. Data Access and Preprocessing: Users can create a knowledge base and upload multimodal data from office scenarios, including emails with attachments, contract PDFs attached to emails, embedded project flowcharts, meeting voice messages, and online meeting recordings. The system automatically verifies the format validity and whether it is corrupted. Among them, emails are text, contract PDFs contain complex tables, flowcharts are images, meeting voice messages are audio, and online meeting recordings are video. Step 2. In-depth analysis and modal text conversion: The system's different modal text conversion frameworks, except for DeepDoc which is loaded from the local directory, all use a large model service that is compatible with the openai-api format and is started by the vllm framework. Different base large models are used to process different modalities, including document parsing, image parsing, video parsing and audio parsing. Step 3. In-depth analysis and modal text conversion: Unified representation of heterogeneous data: The text information from document parsing, the image semantics from image parsing, the temporal text from video parsing, and the speech content from audio parsing are converted into vectors in the same semantic space using a unified BGE-Large-ZH-V1.5 vector model, so that data from different modalities can form associated representations in a unified semantic space. BGE-Large-ZH-V1.5 is a large model service compatible with the OpenAI API format, launched using the VLLM framework. The core principle of the BGE-Large-ZH-V1.5 model is based on embedding technology, which converts text into a high-dimensional vector representation. In this way, the model can capture the semantic information of the text, thereby achieving efficient text retrieval and semantic similarity calculation. Specifically, the model learns the semantic representation of the text through two stages: pre-training and fine-tuning. The pre-training stage uses a large-scale corpus for training, while the fine-tuning stage optimizes the model according to the specific task requirements. Step 4. Construction of the "keyword-vector" dual index system: Keyword retrieval directly scans the inverted index of the original text to ensure accurate matching of technical terms; vector retrieval relies on a unified representation vector to capture deep semantic relationships; dual-channel parallel processing achieves the synergistic effect of "hard matching" and "soft association"; Step 5. Cross-modal retrieval: The user inputs a search request: uploading an audio clip, such as a voice question or data in any modality; the modality type uploaded by the user is identified, and after identifying the request as an audio modality, the SenseVoice model first extracts key information, such as semantic content or sentiment, and converts the audio into text. Then, the text is converted into a vector representation as a search vector, and keywords are extracted as the core basis for the search; the user adjusts the weights of vector search and keyword search based on the characteristics of the audio modality and their own needs to perform a hybrid search, where keyword search is based on the original text stored in the database.

[0015] Furthermore, the document parsing process in step 2 is as follows: The document is fully parsed using the DeepDoc model, identifying and distinguishing various layout components, including text, titles, tables, and image titles. The table structure is extracted in a refined manner, including columns, rows, column headers, row headers, and merged cell tags. The table content is converted into structured data in a format such as JSON, while continuous text blocks and associated title information are extracted.

[0016] Furthermore, the image parsing process in step 2 is as follows: The InternVL-Chat model is used to deeply analyze the elements, structure, and semantic information in the image to identify surface visual elements, such as objects or scenes, and extract deep semantic information, such as action intentions and emotional tendencies, to generate text descriptions that conform to natural language logic and include emotional semantic tendencies.

[0017] Furthermore, the video parsing process in step 2 is as follows: A two-layer processing logic of "inter-frame difference analysis + semantic conversion" is adopted. First, accurate keyframe extraction is achieved based on OpenCV technology. By calculating the pixel differences between consecutive frames, key nodes of scene transitions are automatically identified, and the keyframes with the strongest semantic representativeness are selected. Then, the extracted keyframes are input into the InternVL-Chat model for deep semantic parsing, identifying visual elements in the scene. Visual elements in the scene, such as chart types or human actions, generate logically consistent text descriptions, achieving accurate conversion of video content from visual signals to text semantics, providing a relatable text foundation for subsequent cross-modal retrieval. Pixel differences between consecutive frames include changes in motion trajectory, color distribution shifts, and changes in object outlines. The keyframes with the strongest semantic representativeness require the removal of duplicate redundant frames and blurred transition frames.

[0018] Furthermore, the audio parsing process in step 2 is as follows: the recording is parsed in multiple layers using the SenseVoice model, and the audio is accurately converted into text using speech-to-text technology, providing multi-dimensional audio feature support for cross-modal retrieval.

[0019] Furthermore, the core logic of the hybrid retrieval in step 5 is a dynamic combination of keyword retrieval and vector retrieval kNN, as follows: Keyword retrieval: First, a basic query is constructed. A basic query object, `bqry`, is generated based on the modal-transformed text uploaded by the user. Then, filtering conditions are dynamically added according to the request parameters. Finally, the query is executed, applying `bqry` to the Elasticsearch query and performing text keyword matching based on the Elasticsearch inverted index, while adhering to the following filtering logic. 1) Precise filtering: Based on the kb_ids parameter of the knowledge base ID and the doc_ids parameter of the document ID, use terms query to return only documents with the specified ID; 2) Range filtering: For numeric fields such as available_int, use range queries to filter documents that match the numeric range; 3) Logical filtering: Use boolean queries with clauses such as must_not to exclude documents that do not meet the conditions.

[0020] Furthermore, the vector retrieval process is as follows: If the request includes a vector parameter, an embedding model such as BGE-Large-ZH-V1.5 is used to convert the user's question into a vector q_vec, and a kNN query is constructed. During execution, the returned results are filtered based on a similarity threshold and a top-k limit on the number of returned results. Finally, the keyword search filtering condition bqry is applied to the vector search to ensure that the vector search results meet the keyword filtering conditions. If both keyword and vector search conditions exist, the vector search kNN is executed first, and results that meet the keyword filtering conditions are filtered in the vector space. The keyword search highlighting function only works when vector search is not enabled; the highlight parameter is removed when vector search is enabled. The vector similarity calculation method used in vector search is as follows: Where A*B is the dot product of vectors A and B; ||A|| and ||B|| are the L2 norms of the two vectors respectively; the vector similarity calculation results are in the range of [−1,1], and the closer the value is to 1, the more similar the vectors are in direction.

[0021] The present invention provides a method system for cross-modal mutual inspection of graphics, text, audio and video based on deep understanding and heterogeneous fusion, which includes a module for executing the method for cross-modal mutual inspection of graphics, text, audio and video based on deep understanding and heterogeneous fusion.

[0022] The beneficial effects of this invention are: This invention transforms the parsing results of documents, images, videos, and audio into vectors of a unified dimension, strengthening the semantic association of multimodal vectors, bridging the semantic gap between different modalities, improving the accuracy of cross-modal retrieval, and avoiding the omission of relevant results. It adopts a parallel query with "keyword-vector" dual indexes and a dynamic weighting strategy to accelerate the retrieval response speed, and can be widely adapted to the needs of rapid information location in multiple fields such as office scenarios, government services, medical fields, and education scenarios.

[0023] This invention first converts multimodal data such as documents, images, audio, and video into text content: documents undergo deep document parsing and semantic understanding using DeepDoc; images and videos are described using the InternVL-Chat model; audio is transcribed and extracted using the SenseVoice model, followed by deep semantic parsing; then, a unified BGE-Large-ZH-V1.5 vector model is used to convert the text content into feature vectors in a unified semantic space, thereby achieving the construction of cross-modal semantic associations. It strengthens cross-modal semantic consistency: through the dual transformation of "text mediation + unified vectors," the visual features of images, such as color and composition, the acoustic features of audio, such as tone and rhythm, and the temporal features of videos, such as scene transitions, are all mapped to the same semantic space. This allows descriptions of "experiment failure" in documents, "instrument error" in videos, and "operational error" in audio to be directly related through vector distance, fundamentally solving the "semantic fragmentation" problem caused by differences in feature dimensions between different modalities.

[0024] This invention's module is divided into two layers: parser and vision. The parser defines parsing methods for different data source types, with special handling for resumes. The vision defines visual tasks for PDF documents. The model types employ the lightweight ONNX format VIT model, including the DET model for text detection, the REC model for text recognition, the Layout model for layout recognition (including sub-models for laws, manuals, and papers), and the TSR model for table recognition. The parser types include parsers for various document formats such as docx and excel. The PDF parser will call different layout recognition models according to categories, and a regular parser is also provided. This invention's deep document understanding module uses the DeepDo model, capable of extracting insights from unstructured data of various complex formats. It can truly complete needle-in-a-haystack tests quickly in scenarios with unlimited context tokens. For user-uploaded documents, the system can automatically identify the document layout, including titles, paragraphs, line breaks, etc. For tables, it can identify the presence of tables in the document and identify the table layout, including each cell within the table and whether multiple lines of text need to be merged into one cell. Finally, the table content is processed in conjunction with the header information to ensure it is sent to the database in an appropriate format, providing accurate knowledge blocks for retrieval. Meanwhile, DeepDoc is highly adaptable to parsing different document formats, especially for extracting fine-grained information from complex resumes; its small model and excellent recognition and extraction performance enable it to efficiently handle document parsing tasks.

[0025] This invention manually adjusts the weights of keyword retrieval and vector retrieval based on the input modality, such as text, image, and audio. Vector retrieval is based on a unified multimodal text vector, forming a keyword-vector dual hybrid retrieval mechanism. Keyword retrieval ensures accurate matching of professional terms, specific names, etc., such as legal article numbers and product models. Vector retrieval captures semantic relationships, such as synonyms and similar content across expressions, avoiding the limitations of a single retrieval method and balancing accuracy and generalization.

[0026] This invention boasts strong scalability. Text is the most universal information transmission method between different modalities. The unified representation mechanism is compatible with new modalities, such as 3D models and sensor data, through text as a universal carrier. It does not rely on specific modal-specific feature extractors, such as traditional image retrieval relying on SIFT features and text retrieval relying on TF-IDF. Instead, it only needs to convert the new modality into a text description to be incorporated into the existing vector system, significantly reducing the adaptation cost of adding new modalities and supporting the smooth expansion of the system from "processing 4 modalities" to "N modalities". With business development and technological innovation, new data modalities and application scenarios are constantly emerging. Thanks to its strong scalability, the system can easily accommodate new functional modules and data types. For example, when it is necessary to access geographic information modal data, a corresponding conversion interface can be developed to convert the geographic information data into text form and seamlessly integrate it into the existing system architecture, greatly expanding the application boundaries and scope of the system.

[0027] This invention reduces the technical complexity of cross-modal retrieval. Compared with directly constructing pairwise mapping relationships such as "image-audio" and "video-text", the unified representation simplifies the cross-modal association problem to "similarity calculation between text vectors" through a two-step transformation of "multimodal → text → vector", reducing model design and training costs, while improving the maintainability and stability of the system.

[0028] The retrieval mechanism of this invention is highly flexible and can adapt to diverse scenario needs compared to fixed weight strategies. For example, legal document retrieval requires enhanced keyword matching accuracy, while creative design retrieval requires emphasis on semantic association, greatly improving personalized retrieval satisfaction. Detailed Implementation

[0029] The present invention provides a cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion, comprising the following steps: Step 1. Data Access and Preprocessing: Users can create a knowledge base and upload multimodal data from office scenarios, including emails with attachments, contract PDFs attached to emails, embedded project flowcharts, meeting voice messages, and online meeting recordings. The system automatically verifies the format validity and whether it is corrupted. Among them, emails are text, contract PDFs contain complex tables, flowcharts are images, meeting voice messages are audio, and online meeting recordings are video. Step 2. In-depth analysis and modal text conversion: The system's different modal text conversion frameworks, except for DeepDoc which is loaded from the local directory, all use a large model service that is compatible with the openai-api format and is started by the vllm framework. Different base large models are used to process different modalities, including document parsing, image parsing, video parsing and audio parsing. The document parsing process is as follows: The document is fully parsed using the DeepDoc model, which identifies and distinguishes various layout components, including text, headings, tables, and image titles. The table structure is extracted in a refined manner, including columns, rows, column headings, row headings, and merged cell labels. The table content is converted into structured data in a format such as JSON, while continuous text blocks and related heading information are extracted. The image parsing process is as follows: The InternVL-Chat model is used to deeply analyze the elements, structure and semantic information in the image to identify surface visual elements, such as objects or scenes, and extract deep semantic information, such as action intentions and emotional tendencies, to generate text descriptions that conform to the logic of natural language and include emotional semantic tendencies, such as "Image shows: In a laboratory scene, people are operating precision instruments, and the background screen displays data curves"; The video analysis process is as follows: A two-layer processing logic of "inter-frame difference analysis + semantic conversion" is adopted. First, based on OpenCV technology, accurate keyframe extraction is achieved. By calculating the pixel differences between consecutive frames, key nodes of scene transitions are automatically identified, and the keyframes with the strongest semantic representativeness are selected. Then, the extracted keyframes are input into the InternVL-Chat model for deep semantic analysis, identifying visual elements in the scene, such as chart types or human actions, and generating logical text descriptions. This achieves accurate conversion of video content from visual signals to text semantics, providing a relatable text foundation for subsequent cross-modal retrieval. Pixel differences between consecutive frames include changes in motion trajectory, color distribution shifts, and object outline changes. The keyframes with the strongest semantic representativeness require the removal of duplicate and redundant frames and blurred transition frames. Key nodes of scene transitions are automatically identified, such as when the scene jumps from "background introduction" to "core process demonstration." Logical text descriptions are generated, such as "Frame 5: Showing the product architecture diagram, highlighting core modules → Frame 12: Switching to operation step demonstration, arrow pointing to key button." The parsing process is as follows: The recording is parsed in multiple layers using the SenseVoice model, and the audio is accurately converted into text using speech-to-text technology, providing multi-dimensional audio feature support for cross-modal retrieval.

[0030] Step 3. In-depth analysis and modal text conversion: Unified representation of heterogeneous data: The text information from document parsing, the image semantics from image parsing, the temporal text from video parsing, and the speech content from audio parsing are converted into vectors in the same semantic space using a unified BGE-Large-ZH-V1.5 vector model, so that data from different modalities can form associated representations in a unified semantic space. BGE-Large-ZH-V1.5 is a large model service compatible with the OpenAI API format, launched using the VLLM framework. The core principle of the BGE-Large-ZH-V1.5 model is based on embedding technology, which converts text into a high-dimensional vector representation. In this way, the model can capture the semantic information of the text, thereby achieving efficient text retrieval and semantic similarity calculation. Specifically, the model learns the semantic representation of the text through two stages: pre-training and fine-tuning. The pre-training stage uses a large-scale corpus for training, while the fine-tuning stage optimizes the model according to the specific task requirements. The BGE-Large-ZH-V1.5 model employs a variety of advanced technologies and algorithms, including multilingual support: the model supports Chinese and several other languages, enabling it to handle multilingual text retrieval tasks; long text processing: the model can handle inputs of up to 8192 tokens, making it suitable for long text retrieval and analysis; and embedding optimization: by introducing techniques such as instruction fine-tuning and hard negative sample mining, the model can generate a more reasonable similarity distribution, improving retrieval performance.

[0031] Step 4. Construction of the "keyword-vector" dual index system: Keyword retrieval directly scans the inverted index of the original text to ensure accurate matching of technical terms; vector retrieval relies on a unified representation vector to capture deep semantic relationships; dual-channel parallel processing achieves the synergistic effect of "hard matching" and "soft association"; capturing deep semantic relationships, such as mapping "semiconductor lithography process" to a similar embedding space; Step 5. Cross-modal retrieval: The user inputs a search request: uploading an audio clip, such as a voice question or data in any modality; the modality type uploaded by the user is identified, and after identifying the request as an audio modality, the SenseVoice model first extracts key information, such as semantic content or sentiment, and converts the audio into text. Then, the text is converted into a vector representation as a search vector. At the same time, keywords are extracted as the core basis for the search. The user adjusts the weights of vector search and keyword search based on the characteristics of the audio modality and their own needs to perform a mixed search, where keyword search is based on the original text stored in the database. The core logic of hybrid retrieval is a dynamic combination of keyword retrieval and vector retrieval kNN, as follows: Keyword retrieval: First, a basic query is constructed. A basic query object, `bqry`, is generated based on the modal-transformed text uploaded by the user. Then, filtering conditions are dynamically added according to the request parameters. Finally, the query is executed, applying `bqry` to the Elasticsearch query and performing text keyword matching based on the Elasticsearch inverted index, while adhering to the following filtering logic. 1) Precise filtering: Based on the kb_ids parameter of the knowledge base ID and the doc_ids parameter of the document ID, use terms query to return only documents with the specified ID; 2) Range filtering: For numeric fields such as available_int, use range queries to filter documents that match the numeric range; 3) Logical filtering: Use boolean queries with clauses such as must_not to exclude documents that do not meet the criteria; The vector retrieval process is as follows: If the request includes a vector parameter, an embedding model such as BGE-Large-ZH-V1.5 is used to convert the user's question into a vector q_vec, and a kNN query is constructed. During execution, the returned results are filtered based on a similarity threshold and a top-k limit on the number of returned results. Finally, the keyword search filtering condition bqry is applied to the vector search to ensure that the vector search results meet the keyword filtering conditions. If both keyword and vector search conditions exist, the vector search kNN is executed first, and results that meet the keyword filtering conditions are filtered in the vector space. The keyword search highlighting function only works when vector search is not enabled; the highlight parameter is removed when vector search is enabled. The vector similarity calculation method used in vector search is as follows: Where A*B is the dot product of vectors A and B; ||A|| and ||B|| are the L2 norms of the two vectors respectively; the vector similarity calculation results are in the range of [−1,1], and the closer the value is to 1, the more similar the vectors are in direction.

[0032] The overall logic of hybrid retrieval is based on vector retrieval, recalling semantically relevant documents in the vector space while applying keyword filtering conditions to ensure accuracy. If vector retrieval fails, the conditions are relaxed and the retrieval is repeated. This strategy leverages the semantic understanding capabilities of vector retrieval while retaining the precise matching advantage of keyword retrieval, achieving efficient and comprehensive information retrieval. Keyword retrieval excels at precisely matching specific terms, such as legal provisions or product models, while vector retrieval excels at capturing semantic connections, such as synonyms and cross-modal similar content. Combining the two can balance accuracy and generalization.

[0033] The present invention provides a cross-modal mutual inspection system for text, image, audio and video based on deep understanding and heterogeneous data fusion, comprising a module that executes a cross-modal mutual inspection method for text, image, audio and video based on deep document understanding and heterogeneous data fusion.

Claims

1. A cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion, characterized by: The steps include the following: Step 1. Data Access and Preprocessing: Users can create a knowledge base and upload multimodal data from office scenarios, including emails with attachments, contract PDFs attached to emails, embedded project flowcharts, meeting voice messages, and online meeting recordings. The system automatically verifies the format validity and whether it is corrupted. Among them, emails are text, contract PDFs contain complex tables, flowcharts are images, meeting voice messages are audio, and online meeting recordings are video. Step 2. In-depth analysis and modal text conversion: The system's different modal text conversion frameworks, except for DeepDoc which is loaded from the local directory, all use a large model service that is compatible with the openai-api format and is started by the vllm framework. Different base large models are used to process different modalities, including document parsing, image parsing, video parsing and audio parsing. Step 3. In-depth analysis and modal text conversion: Unified representation of heterogeneous data: The text information from document parsing, the image semantics from image parsing, the temporal text from video parsing, and the speech content from audio parsing are converted into vectors in the same semantic space using a unified BGE-Large-ZH-V1.5 vector model, so that data from different modalities can form associated representations in a unified semantic space. BGE-Large-ZH-V1.5 is a large model service compatible with the OpenAI API format, launched using the VLLM framework. The core principle of the BGE-Large-ZH-V1.5 model is based on embedding technology, which converts text into a high-dimensional vector representation. In this way, the model can capture the semantic information of the text, thereby achieving efficient text retrieval and semantic similarity calculation. Specifically, the model learns the semantic representation of the text through two stages: pre-training and fine-tuning. The pre-training stage uses a large-scale corpus for training, while the fine-tuning stage optimizes the model according to the specific task requirements. Step 4: Construction of the "keyword-vector" dual indexing system: Keyword retrieval directly scans the inverted index of the original text to ensure accurate matching of technical terms; vector retrieval relies on a unified representation vector to capture deep semantic relationships; dual-channel parallel processing achieves the synergistic effect of "hard matching" and "soft association"; Step 5. Cross-modal retrieval: The user inputs a search request: uploading an audio clip, such as a voice question or data in any modality; the modality type uploaded by the user is identified, and after identifying the request as an audio modality, the SenseVoice model first extracts key information, such as semantic content or sentiment, and converts the audio into text. Then, the text is converted into a vector representation as a search vector, and keywords are extracted as the core basis for the search; the user adjusts the weights of vector search and keyword search based on the characteristics of the audio modality and their own needs to perform a hybrid search, where keyword search is based on the original text stored in the database.

2. The cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion as described in claim 1, characterized in that: The document parsing process in step 2 is as follows: The document is fully parsed using the DeepDoc model, and various layout components are identified and distinguished. These layout components include text, titles, tables, and image titles. The table structure is extracted in a refined manner, including columns, rows, column headers, row headers, and merged cell tags. The table content is converted into structured data in a format such as JSON, and continuous text blocks and associated title information are extracted.

3. The cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion as described in claim 1, characterized in that: The image parsing process in step 2 is as follows: The InternVL-Chat model is used to deeply analyze the elements, structure, and semantic information in the image to identify surface visual elements, such as objects or scenes, and extract deep semantic information, such as action intentions and emotional tendencies, to generate text descriptions that conform to natural language logic and include emotional semantic tendencies.

4. The cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion as described in claim 1, characterized in that: The video parsing process in step 2 is as follows: A two-layer processing logic of "inter-frame difference analysis + semantic conversion" is adopted. First, accurate keyframe extraction is achieved based on OpenCV technology. By calculating the pixel differences between consecutive frames, key nodes of scene transitions are automatically identified, and the keyframes with the strongest semantic representativeness are selected. Then, the extracted keyframes are input into the InternVL-Chat model for deep semantic parsing, identifying visual elements in the scene. Visual elements in the scene, such as chart types or human actions, generate logically consistent text descriptions, achieving accurate conversion of video content from visual signals to text semantics, providing a relatable text foundation for subsequent cross-modal retrieval. Pixel differences between consecutive frames include changes in motion trajectory, color distribution shifts, and object contour changes. The keyframes with the strongest semantic representativeness require the removal of duplicate and redundant frames and blurred transition frames.

5. The cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion as described in claim 1, characterized in that: The audio parsing process in step 2 is as follows: the recording is parsed in multiple layers using the SenseVoice model, and the audio is accurately converted into text using speech-to-text technology, providing multi-dimensional audio feature support for cross-modal retrieval.

6. The cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion as described in claim 1, characterized in that: The core logic of hybrid retrieval in step 5 is the dynamic combination of keyword retrieval and vector retrieval kNN, as follows: Keyword retrieval: First, a basic query is constructed. A basic query object, `bqry`, is generated based on the modal-transformed text uploaded by the user. Then, filtering conditions are dynamically added according to the request parameters. Finally, the query is executed, applying `bqry` to the Elasticsearch query and performing text keyword matching based on the Elasticsearch inverted index, while adhering to the following filtering logic. 1) Precise filtering: Based on the kb_ids parameter of the knowledge base ID and the doc_ids parameter of the document ID, use terms query to return only documents with the specified ID; 2) Range filtering: For numeric fields such as available_int, use range queries to filter documents that match the numeric range; 3) Logical filtering: Use boolean queries with clauses such as must_not to exclude documents that do not meet the conditions.

7. The cross-modal mutual inspection method for graphics, text, audio, and video based on deep understanding and heterogeneous fusion as described in claim 6, characterized in that: The vector retrieval process is as follows: If the request includes a vector parameter, an embedding model such as BGE-Large-ZH-V1.5 is used to convert the user's question into a vector q_vec, and a kNN query is constructed. During execution, the returned results are filtered based on a similarity threshold and a top-k limit on the number of returned results. Finally, the keyword search filtering condition bqry is applied to the vector search to ensure that the vector search results meet the keyword filtering conditions. If both keyword and vector search conditions exist, the vector search kNN is executed first, and results that meet the keyword filtering conditions are filtered in the vector space. The keyword search highlighting function only works when vector search is not enabled; the highlight parameter is removed when vector search is enabled. The vector similarity calculation method used in vector search is as follows: Where A*B is the dot product of vectors A and B; ||A|| and ||B|| are the L2 norms of the two vectors respectively; the vector similarity calculation results are in the range of [−1,1], and the closer the value is to 1, the more similar the vectors are in direction.

8. A cross-modal mutual inspection method system for graphics, text, audio, and video based on deep understanding and heterogeneous fusion, characterized by: The system includes a module for performing a cross-modal mutual inspection method for text, image, audio and video based on deep document understanding and heterogeneous data fusion as described in any one of claims 1-7.

Images