Multi-modal retrieval traceability method and device and computing equipment
By constructing a multimodal retrieval and tracing method, establishing a mapping relationship between metadata and feature vectors, and combining a two-stage hybrid retrieval mechanism, the heterogeneity and tracing problems of multimodal data are solved, achieving efficient, reliable cross-modal retrieval and interpretability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-09
- Publication Date
- 2026-05-08
AI Technical Summary
In existing technologies, the construction and retrieval of knowledge bases for multimodal data suffer from modal heterogeneity, challenges in efficient indexing and retrieval, as well as issues of traceability and interpretability. It is difficult to achieve cross-modal alignment and association, and it lacks credibility and interpretability.
A first vector library and a second vector library are constructed. The mapping relationship between the metadata and semantic text of the original file and the feature vector and metadata is established respectively. The query vector is obtained and the query results and traceability information are generated through a two-stage hybrid retrieval mechanism.
It achieves efficient cross-modal retrieval, provides a complete traceability chain, ensures the credibility and interpretability of retrieval results, and improves the scalability and maintainability of the knowledge base.
Smart Images
Figure CN121996807A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal retrieval technology, and in particular to a multimodal retrieval tracing method, a multimodal retrieval tracing device, and a computing device. Background Technology
[0002] Multimodal data typically includes text, image, audio, and video data. Text data usually consists of character sequences, words, or sentences; image data is generally represented by pixels and their three-channel color values; audio data is primarily represented by time-series waveforms or spectrograms; video data consists of consecutive frames of images and their accompanying audio signals, containing both spatial and temporal information. The primary responsibility of a multimodal knowledge base system is to uniformly store and efficiently retrieve this diverse range of multimodal data.
[0003] Currently, the following problems exist in the construction and retrieval of knowledge bases for different modalities of data: 1) Modal heterogeneity problem: Different modal data have different representation methods, and it is difficult to construct a unified representation space and achieve cross-modal alignment and association.
[0004] 2) Efficient indexing and retrieval issues: Faced with massive multimodal data, how to design a hybrid retrieval mechanism that ensures both high recall and high accuracy is an urgent problem to be solved.
[0005] 3) Traceability and interpretability issues: In the process of cross-modal retrieval and reasoning, it is necessary to ensure that each result can be traced back to its original data source, and at the same time provide an interpretable chain of evidence to enhance the credibility and auditability of the system.
[0006] Therefore, a multimodal retrieval and tracing method is needed to solve the problems existing in the above-mentioned technical solutions. Summary of the Invention
[0007] Therefore, the present invention provides a multimodal retrieval and tracing method and apparatus to solve or at least alleviate the problems mentioned above.
[0008] According to one aspect of the present invention, a multimodal retrieval and tracing method is provided, executed in a computing device, the computing device including a first vector library and a second vector library constructed based on multiple original files of multiple modalities, wherein the first vector library is adapted to establish a mapping relationship between the metadata and semantic text of each original file and the corresponding vector representation, and the second vector library is adapted to establish a mapping relationship between the feature vector of each original file and the metadata of the original file; the method includes: obtaining a query question; vectorizing the query question to obtain a corresponding query vector; based on the query vector, obtaining corresponding target metadata and target text semantics from the first vector library; based on the target metadata, obtaining corresponding target feature vectors from the second vector library, and performing similarity matching in the second vector library based on the target feature vectors to obtain multiple related feature vectors from multiple original files of multiple modalities, and obtaining related metadata corresponding to each related feature vector; and generating query results and tracing information corresponding to the query question based on the target metadata, the target semantic text, and the related metadata corresponding to each related feature vector.
[0009] Optionally, the multimodal retrieval and tracing method according to the present invention further includes: extracting semantic text from each of the original files and obtaining metadata of the original files, wherein the semantic text is used to describe the content of the original files; generating a table file containing multiple metadata entries based on the metadata and semantic text of multiple original files of multiple modalities, wherein each metadata entry contains metadata and semantic text of an original file, and the metadata includes file path, file size, and file type; performing vectorization processing on each metadata entry to obtain a corresponding vector representation, and storing the vector representation in a first vector library in association with the metadata entry; extracting feature vectors from each of the original files and storing the feature vectors in a second vector library in association with the metadata of the original files.
[0010] Optionally, in the multimodal retrieval and tracing method according to the present invention, extracting feature vectors from each of the original files and storing the feature vectors in a second vector library in association with the metadata of the original files includes: extracting feature vectors from each of the original files and storing the feature vectors in a second vector library in association with the file paths in the metadata of the original files.
[0011] Optionally, in the multimodal retrieval and tracing method according to the present invention, obtaining the corresponding target metadata and target text semantics from the first vector library based on the query vector includes: obtaining the corresponding target metadata entry from the first vector library based on the query vector, wherein the target metadata entry contains target metadata and target text semantics.
[0012] Optionally, in the multimodal retrieval and tracing method according to the present invention, generating query results and tracing information corresponding to the query question based on the target metadata, the target semantic text, and the relevant metadata corresponding to each of the relevant feature vectors includes: using a large language model, generating query results and tracing information corresponding to the query question based on the query question, the target metadata entries, and the relevant metadata corresponding to each of the relevant feature vectors; wherein, the query results include the target original file corresponding to the target metadata; the tracing information includes the file path of the target original file, the target metadata entries corresponding to the target metadata, the similarity calculation results in the first vector library and the second vector library, and the original file information corresponding to each of the relevant metadata.
[0013] Optionally, in the multimodal retrieval and tracing method according to the present invention, obtaining the corresponding target metadata and target text semantics from the first vector library based on the query vector includes: calculating a first similarity between the query vector and each vector representation in the first vector library to obtain one or more target vector representations with the highest first similarity, and obtaining the target metadata and target text semantics corresponding to each target vector representation; obtaining the corresponding target feature vector from the second vector library based on the target metadata, and performing similarity matching in the second vector library based on the target feature vector to obtain multiple related feature vectors from multiple original files of multiple modalities, including: matching the target metadata with each metadata in the second vector library to obtain the corresponding target feature vector, and calculating a second similarity between the target feature vector and each feature vector in the second vector library to obtain multiple related feature vectors from multiple original files of multiple modalities with the highest second similarity.
[0014] Optionally, in the multimodal retrieval and tracing method according to the present invention, the multiple original files of the multiple modalities include text files, image files, audio files, and video files; feature vectors are extracted from each of the original files, including: extracting text feature vectors from text files using a pre-trained language model; extracting image feature vectors from image files using a visual Transformer model; extracting audio feature vectors from audio files using an audio spectrum analysis model; and extracting video feature vectors from video files using a video frame sequence analysis model and an audio spectrum analysis model.
[0015] According to one aspect of the present invention, a multimodal retrieval and tracing apparatus is provided, deployed in a computing device, suitable for performing the method described above. The computing device includes a first vector library and a second vector library constructed based on multiple original files of various modalities. The first vector library is adapted to establish a mapping relationship between the metadata and semantic text of each original file and its corresponding vector representation. The second vector library is adapted to establish a mapping relationship between the feature vector of each original file and the metadata of the original file. The apparatus includes: The vectorization processing module is suitable for obtaining the query question and performing vectorization processing on the query question to obtain the corresponding query vector; The first retrieval module is adapted to obtain the corresponding target metadata and target text semantics from the first vector library based on the query vector; The second retrieval module is adapted to obtain the corresponding target feature vector from the second vector library based on the target metadata, and to perform similarity matching in the second vector library based on the target feature vector to obtain multiple related feature vectors from multiple original files of multiple modalities, and to obtain the related metadata corresponding to each of the related feature vectors; The generation module is adapted to generate query results and source information corresponding to the query question based on the target metadata, the target semantic text, and the relevant metadata corresponding to each of the relevant feature vectors.
[0016] According to one aspect of the present invention, a computing device is provided, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions including instructions for performing the multimodal retrieval and tracing method as described above.
[0017] According to one aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method as described above.
[0018] According to one aspect of the present invention, a readable storage medium storing program instructions is provided, which, when read and executed by a computing device, causes the computing device to perform the multimodal retrieval and tracing method as described above.
[0019] According to the technical solution of the present invention, a multimodal retrieval and source tracing method is provided. A first vector library and a second vector library are pre-constructed based on multiple original files of various modalities. The first vector library establishes a mapping relationship between the metadata and semantic text of each original file and its corresponding vector representation. The second vector library establishes a mapping relationship between the feature vectors of each original file and its metadata. When a query question is obtained, it is vectorized into a corresponding query vector. Based on the query vector, the corresponding target metadata and target text semantics are obtained from the first vector library. Then, based on the target metadata, the corresponding target feature vector is obtained from the second vector library. Similarity matching is performed on the target feature vectors in the second vector library to obtain multiple relevant feature vectors from multiple original files of various modalities. The relevant metadata corresponding to each relevant feature vector is also obtained. Finally, query results and source tracing information corresponding to the query question are generated. In this way, establishing the association between the two vector libraries based on metadata, combined with a two-stage hybrid retrieval mechanism, enables efficient cross-modal retrieval. Moreover, it provides a complete source tracing link for the retrieval results, ensuring the credibility and interpretability of the retrieval results.
[0020] Furthermore, by uniformly converting the metadata and semantic text of raw files from different modalities into a tabular structure, a standardized representation of heterogeneous multi-modal file data can be achieved. Moreover, vectorizing each metadata entry facilitates the addition of new data and the updating of existing data, thereby improving the scalability and maintainability of the knowledge base.
[0021] Furthermore, based on the characteristics of different modal data such as text, images, audio, and video, corresponding dedicated encoding models are used for vectorization processing, so that they can be represented in a unified vector space. This enables semantic alignment and association of cross-modal data.
[0022] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description
[0023] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The foregoing and other objectives, features, and advantages of the invention will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. Throughout the invention, the same reference numerals generally refer to the same parts or elements.
[0024] Figure 1 A schematic diagram of a computing device 100 provided according to an embodiment of the present invention is shown; Figure 2 A flowchart illustrating a multimodal retrieval and tracing method 200 provided according to an embodiment of the present invention is shown; Figure 3 A schematic diagram of a multimodal retrieval and tracing device 300 provided according to an embodiment of the present invention is shown. Detailed Implementation
[0025] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0026] To address the current problems in knowledge base construction and retrieval for data of different modalities, this invention proposes a multimodal retrieval and tracing method. It pre-constructs a first vector library and a second vector library based on multiple original files of various modalities, and establishes the association between the two vector libraries based on metadata. Combined with a two-stage hybrid retrieval mechanism, it can achieve efficient cross-modal retrieval. Moreover, it can provide a complete tracing link for the retrieval results, ensuring the credibility and interpretability of the retrieval results.
[0027] It should be noted that the multimodal retrieval and tracing method according to the embodiments of the present invention supports cross-modal alignment of large multimodal models.
[0028] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0029] Figure 1 A schematic diagram of a computing device 100 according to an embodiment of the present invention is shown. Figure 1As shown, in a basic configuration, computing device 100 includes at least one processing unit 102 and system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 may be implemented as a processor. System memory 104 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 104 includes an operating system 105.
[0030] According to one aspect, operating system 105 is, for example, suitable for controlling the operation of computing device 100. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. Figure 1 The basic configuration is illustrated by the components within the dashed lines. According to one aspect, the computing device 100 has additional features or functions. For example, according to one aspect, the computing device 100 includes additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 1 The middle part is shown by removable storage device 109 and non-removable storage device 110.
[0031] As stated above, according to one aspect, program module 103 is stored in system memory 104. According to one aspect, program module 103 may include one or more applications. The present invention does not limit the type of application; for example, applications may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slideshow applications, drawing or computer-aided applications, web browser applications, etc.
[0032] According to one aspect, program module 103 may include a plurality of program instructions adapted to execute the multimodal retrieval and tracing method 200 of the present invention, such that computing device 100 is configured to execute the multimodal retrieval and tracing method 200 of the present invention.
[0033] According to one aspect, program module 103 may include a multimodal retrieval and tracing device 300, which may be configured to execute the multimodal retrieval and tracing method 200 of the present invention.
[0034] According to one aspect, examples can be practiced on circuits including discrete electronic components, packaged or integrated electronic chips containing logic gates, circuits utilizing microprocessors, or on a single chip containing electronic components or a microprocessor. For example, it can be practiced via wherein... Figure 1Each or many of the components shown can be implemented as an example by integrating a System-on-a-Chip (SOC) on a single integrated circuit. According to one aspect, such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all integrated (or “burned in”) as a single integrated circuit onto a chip substrate. When operating via the SOC, the functions described herein can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 100. Embodiments of the invention can also be implemented using other techniques capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. Additionally, embodiments of the invention can be implemented within a general-purpose computer or in any other circuit or system.
[0035] According to one aspect, computing device 100 may also have one or more input devices 112, such as a keyboard, mouse, pen, voice input device, touch input device, etc. It may also include output devices 114, such as a display, speaker, printer, etc. The foregoing devices are examples and other devices may also be used. Computing device 100 may include one or more communication connections 116 that allow communication with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver and / or transceiver circuitry; Universal Serial Bus (USB), parallel and / or serial ports.
[0036] As used herein, the term computer-readable medium includes computer storage medium. Computer storage medium can include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (e.g., computer-readable instructions, data structures, or program module 103). System memory 104, removable storage device 109, and non-removable storage device 110 are examples of computer storage media (i.e., memory storage). Computer storage media can include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other article of manufacture that can be used to store information and is accessible by computing device 100. According to one aspect, any such computer storage medium can be part of computing device 100. Computer storage media does not include carrier waves or other transmitted data signals.
[0037] According to one aspect, the communication medium is implemented by computer-readable instructions, data structures, program modules 103, or other data in a modulated data signal (e.g., a carrier wave or other transmission mechanism), and includes any information transmission medium. According to one aspect, the term "modulated data signal" describes a signal having one or more sets of characteristics or altered in a manner that encodes information in the signal. By way of example and not limitation, the communication medium includes wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.
[0038] In an embodiment of the present invention, a computing device 100 is configured to execute the multimodal retrieval and tracing method 200 of the present invention. The computing device 100 includes one or more processors and one or more readable storage media storing program instructions, which, when configured to be executed by the one or more processors, cause the computing device to execute the multimodal retrieval and tracing method 200 of the present invention.
[0039] Figure 2 A flowchart illustrating a multimodal retrieval and tracing method 200 according to an embodiment of the present invention is shown. The multimodal retrieval and tracing method 200 can be executed in a computing device (e.g., the aforementioned computing device 100). The multimodal retrieval and tracing method 200 according to an embodiment of the present invention supports cross-modal alignment of large multimodal models.
[0040] In an embodiment of the present invention, the computing device 100 for executing the multimodal retrieval and tracing method 200 of the present invention may be a server.
[0041] It should be noted that before executing the multimodal retrieval and tracing method 200 in this embodiment of the invention, the computing device 100 may pre-construct a first vector library and a second vector library based on multiple original files of multiple modalities. In other words, the computing device 100 includes a first vector library and a second vector library constructed based on multiple original files of multiple modalities. The first vector library is used to establish the mapping relationship between the metadata and semantic text of each original file and the corresponding vector representation (i.e., the vector representation corresponding to the metadata and semantic text). The second vector library is used to establish the mapping relationship between the feature vector of each original file and the metadata (e.g., file path) of the original file.
[0042] In other words, the first vector library includes metadata and semantic text for each of the multiple original files across multiple modalities, as well as vector representations associated with the metadata and semantic text. The second vector library includes feature vectors for each of the multiple original files across multiple modalities, as well as metadata for the original files associated with the feature vectors. Based on this, a relationship can be established between the first and second vector libraries using metadata.
[0043] like Figure 2 As shown, the multimodal retrieval and tracing method 200 includes the following steps 210-240.
[0044] Step 210: The computing device 100 can obtain the query question and perform vectorization processing on the query question to obtain the corresponding query vector.
[0045] In some embodiments, the computing device 100 can obtain a query question from a client. Specifically, the computing device 100 is communicatively connected to a client, where a user can input a query question. Subsequently, the client can send the user-input query question to the computing device 100. For example, the query question could be "What does the architecture diagram of deep learning look like?".
[0046] In some embodiments, the computing device 100 can vectorize the query question using a text embedding model to obtain the query vector corresponding to the query question.
[0047] Step 220: The computing device 100 can obtain the corresponding target metadata and target text semantics from the first vector library based on the query vector.
[0048] Here, step 220 is the first stage of the retrieval process. The target metadata and target text semantics obtained from the first vector library are the results of the first stage retrieval.
[0049] In some embodiments, in step 220, the computing device 100 first performs similarity matching based on the query vector in the first vector library, that is, calculates the first similarity between the query vector and each vector representation in the first vector library to obtain one or more target vector representations with the highest first similarity (that is, takes one or more vector representations with the highest first similarity to the query vector in the first vector library as one or more target vector representations), and then can obtain the target metadata and target text semantics corresponding to each target vector representation from the first vector library.
[0050] In some embodiments, in step 220, by calculating a first similarity between the query vector and each vector representation in the first vector library, one or more vector representations with a first similarity greater than a first threshold can be obtained as one or more target vector representations. Furthermore, the target metadata and target text semantics corresponding to each target vector representation can be obtained from the first vector library.
[0051] Step 230: Obtain the corresponding target feature vector from the second vector library based on the target metadata (e.g., file path in the target metadata), and perform similarity matching in the second vector library based on the target feature vector to obtain multiple related feature vectors from multiple original files of multiple modalities, and obtain the related metadata corresponding to each related feature vector.
[0052] Here, step 230 is the second-stage retrieval process. The results of the second-stage retrieval include multiple relevant metadata corresponding to multiple relevant feature vectors from multiple original files of various modalities.
[0053] In some embodiments, in step 230, firstly, the target metadata can be matched with each metadata in the second vector library (for example, the file path in the target metadata can be matched with the file path in each metadata in the second vector library) to obtain the corresponding target feature vector. Then, a second similarity can be calculated between the target feature vector and each feature vector in the second vector library to obtain multiple related feature vectors from multiple original files of multiple modalities with the highest second similarity. That is, multiple feature vectors in the second vector library with the highest second similarity to the target feature vector can be used as multiple related feature vectors, and these multiple related feature vectors can come from multiple original files of multiple modalities.
[0054] In some embodiments, in step 230, by calculating the second similarity between the target feature vector and each feature vector in the second vector library, multiple feature vectors (which may come from multiple original files of multiple modalities) with a second similarity greater than a second threshold with the target feature vector can be obtained as multiple related feature vectors.
[0055] Step 240: The computing device 100 can generate query results and source information corresponding to the query question based on the target metadata and target semantic text (first-stage retrieval results) and the relevant metadata corresponding to each relevant feature vector (second-stage retrieval results).
[0056] Therefore, according to the embodiments of the present invention, the association between the first vector library and the second vector library is established based on metadata. Combined with the two-stage hybrid retrieval mechanism, efficient cross-modal retrieval can be achieved. Moreover, it can provide a complete traceability link for the retrieval results, ensuring the credibility and interpretability of the retrieval results.
[0057] In some embodiments, multiple original files of various modalities may include, for example, text files, image files, audio files, and video files.
[0058] In some embodiments, the computing device 100 may pre-construct a first vector library and a second vector library based on multiple original files of various modalities, as detailed below.
[0059] The construction process of the first vector library is as follows: First, semantic text can be extracted from multiple original files of various modalities, that is, semantic text can be extracted from each original file separately. The semantic text describes the content of each original file. Furthermore, metadata for each original file can be obtained. Metadata for an original file may include, for example, the file path, file size, file type, and creation time. In some embodiments, for image files, OCR technology and image understanding models can be used to extract text descriptions and content descriptions from the image file as its semantic text; for audio files, speech recognition technology can be used to convert the audio file into corresponding semantic text; for video files, speech recognition and visual analysis technologies can be combined to extract dialogue content and scene descriptions from the video file as its semantic text. For example, for an image file containing a deep learning architecture diagram, the semantic text extracted from the image file is as follows: "Deep learning neural network architecture diagram, including an input layer, multiple hidden layers, and an output layer."
[0060] Subsequently, based on the metadata and semantic text of multiple original files across various modalities, a tabular file (Excel structured file) containing multiple metadata entries can be generated. Each metadata entry contains the metadata and semantic text of one original file. In this way, the metadata and semantic text of original files from different modalities are uniformly converted into a tabular structure, enabling a standardized representation of heterogeneous multi-modal file data.
[0061] Here is an example of information (metadata and semantic text of the original file) from a metadata entry in a table file: File path: / data / multi_modal / deep_learning.mp4 File size: 156.8MB Creation time: 2024-01-15 10:30:45 File type: Video file Semantic text: Deep learning tutorial videos, including explanations of neural network architecture.
[0062] Furthermore, each metadata entry can be vectorized using a text embedding model to obtain its corresponding vector representation. The vector representations are then associated with the metadata entries and stored in a first vector library (Excel database). This allows for the establishment of a mapping relationship between each vector representation and each metadata entry based on the first vector library.
[0063] For example, the vector representation of the metadata entry in the above example is as follows: [0.234567, -0.123456, 0.456789, ...] In some embodiments, when constructing the second vector library (multimodal vector library), the computing device 100 can extract feature vectors from each original file (each original file among multiple original files of multiple modalities) and associate the feature vectors with the metadata (file path) of the original files and store them in the second vector library. Specifically, the feature vectors extracted from each original file can be associated with the file path in the metadata of the original file and stored in the second vector library. Based on this, the first vector library and the second vector library can establish a precise association relationship based on the file path, and the combination of a two-stage hybrid retrieval mechanism can achieve efficient cross-modal retrieval.
[0064] It should be understood that feature vectors extracted from multiple original files of various modalities may include: text feature vectors corresponding to text files, image feature vectors corresponding to image files, audio feature vectors corresponding to audio files, and video feature vectors corresponding to video files.
[0065] In some embodiments, during the extraction of feature vectors from each original file, for text files, a pre-trained language model can be used to extract text feature vectors (extracting text features and converting them into corresponding text feature vectors). For image files, a visual Transformer model can be used to extract image feature vectors (extracting image features and converting them into corresponding image feature vectors). For audio files, an audio spectrum analysis model can be used to extract audio feature vectors (extracting audio features and converting them into corresponding audio feature vectors). For video files, a video frame sequence analysis model and an audio spectrum analysis model can be used to extract video feature vectors (extracting video features and converting them into corresponding video feature vectors). Based on this, corresponding dedicated encoding models are used for vectorization processing according to the characteristics of different modalities such as text, images, audio, and video, so that they can be represented in a unified vector space. This enables semantic alignment and association of cross-modal data, ensuring the comparability of different modalities of data in the same retrieval system.
[0066] Based on the first vector library constructed according to the above embodiments, in step 220, the corresponding target metadata entry can be obtained from the first vector library based on the query vector. The target metadata entry includes target metadata and target text semantics. Thus, the target metadata and target text semantics corresponding to the query vector can be obtained.
[0067] Specifically, a first similarity can be calculated between the query vector and each vector representation in the first vector library to obtain one or more target vector representations with the highest first similarity (or a first similarity greater than a first threshold). Then, the target metadata entry (containing target metadata and target text semantics) corresponding to each target vector representation can be obtained from the first vector library. For example, the file path in the target metadata of the target metadata entry could be / data / images / deep_learning_arch.jpg.
[0068] Next, in step 230, the file path ( / data / images / deep_learning_arch.jpg) in the target metadata can be matched with the file paths in the metadata of each element in the second vector library to obtain the corresponding target feature vector as an image feature vector. Then, the second similarity between the target feature vector (image feature vector) and each feature vector in the second vector library (including text feature vector, image feature vector, audio feature vector, and video feature vector) can be calculated to obtain multiple related feature vectors from multiple original files of multiple modalities with the highest second similarity (most semantically similar to the target feature vector). These feature vectors may include other image feature vectors, text feature vectors, audio feature vectors, and video feature vectors.
[0069] Thus, according to embodiments of the present invention, semantic association between different modalities of data can be achieved through a unified vector space representation and a two-stage hybrid retrieval mechanism. For example, when a user queries a deep learning architecture diagram, they can not only find related images, but also find related data in multiple modalities such as explanatory videos, technical documents, and code examples through semantic association, thereby achieving cross-modal data association.
[0070] Moreover, the two-stage hybrid retrieval mechanism according to the embodiments of the present invention can significantly improve retrieval efficiency.
[0071] In some embodiments, in step 240, the query question, the target metadata entries obtained in the first retrieval stage, and the relevant metadata (file paths) corresponding to each relevant feature vector obtained in the second retrieval stage can be input into the large language model. The large language model can generate query results and source information corresponding to the query question based on the query question, the target metadata entries, and the relevant metadata corresponding to each relevant feature vector.
[0072] In some embodiments, the query results in step 240 include the target original file corresponding to the target metadata. For example, the query result corresponding to the query question (What does the architecture diagram of deep learning look like?) can be a deep learning architecture diagram.
[0073] Source tracing information includes source tracing links. Specifically, source tracing information (source tracing links) may include the file path of the original target file corresponding to the target metadata (and may also include basic information such as the file size and file type of the original file), the target metadata entries corresponding to the target metadata, the similarity calculation results in the first vector library and the second vector library, and the original file information corresponding to each related metadata (as associated data). Here, the original file information corresponding to the related metadata includes, for example, the file path and file type of the original file. In addition, source tracing information (source tracing links) may also include the vectorization process.
[0074] For example, a source tracing chain can include: the file path of the original target file → target metadata entries → vectorization processing → similarity calculation results → associated data. A complete source tracing chain ensures the credibility and interpretability of the search results. The file path of the original target file is, for example: / data / images / deep_learning_arch.jpg. The target metadata entries are, for example: line 156 of an Excel file, the semantic text "Deep learning neural network architecture diagram...".
[0075] Figure 3 A schematic diagram of a multimodal retrieval and tracing device 300 according to an embodiment of the present invention is shown. The multimodal retrieval and tracing device 300 can be deployed in a computing device 100, and the multimodal retrieval and tracing device 300 is configured to execute the multimodal retrieval and tracing method 200 of the present invention.
[0076] In embodiments of the present invention, the computing device 100 may pre-construct a first vector library and a second vector library based on multiple original files of various modalities. In other words, the computing device 100 includes a first vector library and a second vector library constructed based on multiple original files of various modalities. The first vector library is used to establish a mapping relationship between the metadata and semantic text of each original file and its corresponding vector representation (i.e., the vector representations corresponding to the metadata and semantic text). The second vector library is used to establish a mapping relationship between the feature vector of each original file and the metadata (e.g., file path) of the original file.
[0077] like Figure 3 As shown, in an embodiment of the present invention, the multimodal retrieval and tracing device 300 includes a vectorization processing module 310, a first retrieval module 320, a second retrieval module 330, and a generation module 340 that are sequentially connected in communication.
[0078] The vectorization processing module 310 can obtain the query question and perform vectorization processing on the query question to obtain the corresponding query vector.
[0079] The first retrieval module 320 can obtain the corresponding target metadata and target text semantics from the first vector library based on the query vector.
[0080] The second retrieval module 330 can obtain the corresponding target feature vector from the second vector library based on the target metadata, and perform similarity matching based on the target feature vector in the second vector library to obtain multiple related feature vectors from multiple original files of multiple modalities, and obtain the related metadata corresponding to each related feature vector.
[0081] The generation module 340 can generate query results and source information corresponding to the query question based on the target metadata, target semantic text, and the relevant metadata corresponding to each relevant feature vector.
[0082] It should be noted that the vectorization processing module 310, the first retrieval module 320, the second retrieval module 330, and the generation module 340 are respectively used to execute the aforementioned steps 210 to 240. Here, the specific execution logic of each unit can be found in the description of steps 210 to 240 in the previous method 200, and will not be repeated here.
[0083] According to the multimodal retrieval and tracing method 200 of this embodiment, a first vector library and a second vector library are pre-constructed based on multiple original files of multiple modalities. The first vector library establishes a mapping relationship between the metadata and semantic text of each original file and its corresponding vector representation. The second vector library establishes a mapping relationship between the feature vectors of each original file and its metadata. When a query question is obtained, it is vectorized into a corresponding query vector. Based on the query vector, the corresponding target metadata and target text semantics are obtained from the first vector library. Then, based on the target metadata, the corresponding target feature vector is obtained from the second vector library. Similarity matching is performed on the target feature vectors in the second vector library to obtain multiple relevant feature vectors from multiple original files of multiple modalities. The relevant metadata corresponding to each relevant feature vector is also obtained, ultimately generating query results and tracing information corresponding to the query question. In this way, establishing the association between the two vector libraries based on metadata, combined with a two-stage hybrid retrieval mechanism, enables efficient cross-modal retrieval. Furthermore, it provides a complete tracing link for the retrieval results, ensuring the credibility and interpretability of the retrieval results.
[0084] Furthermore, by uniformly converting the metadata and semantic text of raw files from different modalities into a tabular structure, a standardized representation of heterogeneous multi-modal file data can be achieved. Moreover, vectorizing each metadata entry facilitates the addition of new data and the updating of existing data, thereby improving the scalability and maintainability of the knowledge base.
[0085] Furthermore, based on the characteristics of different modal data such as text, images, audio, and video, corresponding dedicated encoding models are used for vectorization processing, so that they can be represented in a unified vector space. This enables semantic alignment and association of cross-modal data.
[0086] The various techniques described herein can be implemented in combination with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, USB flash drive, floppy disk, CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into and executed by a machine such as a computer, the machine becomes an apparatus for practicing the present invention.
[0087] When the program code is executed on a programmable computer, the mobile terminal generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute the multimodal retrieval and tracing method of the present invention according to instructions in the program code stored in the memory.
[0088] By way of example, and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in the form of modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Any combination of the above is also included within the scope of readable media.
[0089] In the specification provided herein, the algorithms and displays are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with the examples of this invention. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of the invention.
[0090] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0091] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various aspects of the invention, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof.
[0092] Those skilled in the art will understand that modules, units, or components of the devices disclosed in the examples herein can be arranged in the devices described in this embodiment, or alternatively, can be located in one or more devices different from the devices in this example. The modules in the foregoing examples can be combined into a single module or, in addition, can be divided into multiple sub-modules.
[0093] Unless otherwise specified, the use of ordinal numbers such as “first,” “second,” “third,” etc., to describe ordinary objects merely indicates different instances of similar objects and is not intended to imply that the objects being described must have a given order in time, space, ordering, or any other manner.
Claims
1. A multimodal retrieval and tracing method, executed in a computing device, said computing device comprising a first vector library and a second vector library constructed from multiple original files based on multiple modalities, wherein, The first vector library is adapted to establish a mapping relationship between the metadata and semantic text of each original file and its corresponding vector representation; the second vector library is adapted to establish a mapping relationship between the feature vector of each original file and the metadata of the original file; the method includes: Obtain the query question, and perform vectorization processing on the query question to obtain the corresponding query vector; Based on the query vector, the corresponding target metadata and target text semantics are obtained from the first vector library; Based on the target metadata, the corresponding target feature vector is obtained from the second vector library, and similarity matching is performed in the second vector library based on the target feature vector to obtain multiple related feature vectors from multiple original files of multiple modalities, and the related metadata corresponding to each related feature vector is obtained; Based on the target metadata, the target semantic text, and the relevant metadata corresponding to each of the relevant feature vectors, query results and source information corresponding to the query question are generated.
2. The method as described in claim 1, wherein, Also includes: Semantic text is extracted from each of the original files, and metadata of the original files is obtained. The semantic text is used to describe the content of the original files. Based on the metadata and semantic text of multiple original files in various modalities, a table file containing multiple metadata entries is generated, wherein each metadata entry contains the metadata and semantic text of an original file, and the metadata includes file path, file size and file type; Each metadata entry is vectorized to obtain a corresponding vector representation, and the vector representation is associated with the metadata entry and stored in a first vector library; Feature vectors are extracted from each of the original files, and the feature vectors are associated with the metadata of the original files and stored in a second vector library.
3. The method as described in claim 2, wherein, Extracting feature vectors from each of the original files, and associating the feature vectors with the metadata of the original files and storing them in a second vector library, includes: Feature vectors are extracted from each of the original files, and the feature vectors are associated with the file paths in the metadata of the original files and stored in a second vector library.
4. The method as described in claim 2 or 3, wherein, Based on the query vector, the corresponding target metadata and target text semantics are obtained from the first vector library, including: Based on the query vector, the corresponding target metadata entry is obtained from the first vector library. The target metadata entry includes target metadata and target text semantics.
5. The method of claim 4, wherein, Based on the target metadata, the target semantic text, and the relevant metadata corresponding to each of the relevant feature vectors, query results and source information corresponding to the query question are generated, including: Using a large language model, based on the query question, the target metadata entries, and the relevant metadata corresponding to each of the relevant feature vectors, query results and source information corresponding to the query question are generated; The query results include the target original file corresponding to the target metadata; the source information includes the file path of the target original file, the target metadata entry corresponding to the target metadata, the similarity calculation results in the first vector library and the second vector library, and the original file information corresponding to each related metadata.
6. The method according to any one of claims 1-5, wherein, Based on the query vector, the corresponding target metadata and target text semantics are obtained from the first vector library, including: Calculate the first similarity between the query vector and each vector representation in the first vector library to obtain one or more target vector representations with the highest first similarity, and obtain the target metadata and target text semantics corresponding to each target vector representation; Based on the target metadata, the corresponding target feature vector is obtained from the second vector library, and similarity matching is performed in the second vector library based on the target feature vector to obtain multiple relevant feature vectors from multiple original files of multiple modalities, including: The target metadata is matched with each metadata in the second vector library to obtain the corresponding target feature vector, and the second similarity between the target feature vector and each feature vector in the second vector library is calculated to obtain multiple related feature vectors from multiple original files of multiple modalities with the highest second similarity.
7. The method of claim 2, wherein, The multiple original files in various modalities include text files, image files, audio files, and video files; Extracting feature vectors from each of the original files, including: Using a pre-trained language model, extract text feature vectors from text files; Using the visual Transformer model, image feature vectors are extracted from image files; Audio feature vectors are extracted from audio files using an audio spectrum analysis model. Video feature vectors are extracted from video files using video frame sequence analysis models and audio spectrum analysis models.
8. A multimodal retrieval and tracing device, deployed in a computing device, suitable for executing the method as described in any one of claims 1-7, wherein the computing device includes a first vector library and a second vector library constructed based on multiple original files of multiple modalities, wherein, The first vector library is adapted to establish a mapping relationship between the metadata and semantic text of each original file and its corresponding vector representation; the second vector library is adapted to establish a mapping relationship between the feature vector of each original file and the metadata of the original file; the apparatus includes: The vectorization processing module is suitable for obtaining the query question and performing vectorization processing on the query question to obtain the corresponding query vector; The first retrieval module is adapted to obtain the corresponding target metadata and target text semantics from the first vector library based on the query vector; The second retrieval module is adapted to obtain the corresponding target feature vector from the second vector library based on the target metadata, and to perform similarity matching in the second vector library based on the target feature vector to obtain multiple related feature vectors from multiple original files of multiple modalities, and to obtain the related metadata corresponding to each of the related feature vectors; The generation module is adapted to generate query results and source information corresponding to the query question based on the target metadata, the target semantic text, and the relevant metadata corresponding to each of the relevant feature vectors.
9. A computing device, comprising: At least one processor; and A memory storing program instructions, wherein the program instructions are configured to be processed by the at least one processor, the program instructions including instructions for processing the method as described in any one of claims 1-7.
10. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the method as described in any one of claims 1-7.