Multi-modal retrieval system based on similarity between image and text feature vectors

Abstract

A multi-modal retrieval system based on the similarity between image and text feature vectors, which system belongs to the technical field of information retrieval. In the present invention, advanced deep learning technology is used to extract high-dimensional feature vectors of image data and text data, and the similarity between these feature vectors is computed by means of a specific similarity algorithm. The system can process various types of query inputs, which take the form of plain text, plain images and a combination thereof, thereby providing a flexible search mode.

Description

A multimodal retrieval system based on image and text feature vector similarity

[0001] This application claims priority to Chinese Patent Application No. 202410955864.2, filed on July 17, 2024, entitled "A Multimodal Retrieval System Based on Image and Text Feature Vector Similarity", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to the fields of artificial intelligence and information retrieval technology in computer science, and in particular to a multimodal retrieval system based on the similarity of image and text feature vectors. Background Technology

[0003] Currently, it is common to process data in two different modalities: images and text. To achieve effective retrieval, a specific model is needed to transform the features of these two modalities into multimodal feature vectors that can be compared with each other.

[0004] The CLIP (Contrastive Language-Image Pre-training) model, proposed by OpenAI in 2021, is a multimodal pre-trained model based on contrastive learning. It can learn rich cross-modal associations from a large number of image-text pairs and has strong zero-shot learning capabilities. This model is trained by increasing the similarity scores of matched image and text pairs and decreasing the scores of non-matching pairs.

[0005] Faiss (Facebook AI Similarity Search) is an open-source feature vector similarity search engine that efficiently handles indexing and querying of large-scale vector data. Faiss utilizes GPUs (Graphics Processing Units) for accelerated indexing and querying, and supports features such as cluster sharding, read / write separation, and horizontal scaling to adapt to application scenarios with extremely large data volumes.

[0006] The core of similarity measurement methods, especially cross-modal similarity measurement methods, lies in how to accurately measure the similarity between different modalities such as images and text. One approach is to fuse image and text features and then learn a measurement function through a hidden layer; another approach is to use a dual-tower structure to map images and text into a common space and then directly calculate their similarity using methods such as cosine similarity.

[0007] However, improving the efficiency and accuracy of information retrieval remains a challenge in practical applications. Summary of the Invention

[0008] To address the above technical problems, this invention provides a multimodal retrieval system based on the similarity of image and text feature vectors, aiming to improve retrieval efficiency and accuracy by fusing feature vectors from image and text data.

[0009] The technical solution of this invention is:

[0010] A multimodal retrieval system based on the similarity of image and text feature vectors employs advanced deep learning technology to extract high-dimensional feature vectors from image and text data, and calculates the similarity between these feature vectors using a specific similarity algorithm. The system can handle various types of query input, including plain text, plain images, and combinations thereof, thus providing flexible search methods.

[0011] In the image processing module, the system uses the ViT (Vision Transformer) model to analyze images and extract key visual features. Simultaneously, the text processing module employs Natural Language Processing (NLP) technology and the Transformer architecture to understand text content and extract key semantic features.

[0012] The core innovation of the system lies in its multimodal fusion strategy, which considers not only feature associations within a single modality but also feature interactions across modalities. By designing a novel multimodal similarity metric, the system can effectively evaluate and rank the relevance between candidate documents and queries. This approach allows the system to understand the user's query intent in a more granular way and return more accurate search results.

[0013] Furthermore,

[0014] It includes a multimodal feature extraction module, which extracts feature vectors from input information of different modalities; and uses the Transform model to generate vector representations of different modal features in a common feature space.

[0015] The multimodal fusion and alignment module transforms the feature vectors extracted by the multimodal feature extraction module into a unified vector space; it calculates the similarity between the vector representations of different modalities and retrieves relevant cross-modal information from the vector library according to the degree of similarity and returns it to the user;

[0016] Multimodal feature extraction module: includes image feature extraction module and text feature extraction module.

[0017] in,

[0018] Image feature extraction module: Uses the ViT (Vision Transformer) deep learning model to process the input image to extract the image's feature vector;

[0019] The first step of ViT is to divide the image into a series of small patches, each containing P^2 pixels, where P is the preset image patch size;

[0020] In the second step of ViT, after dividing the image into blocks, each block is flattened into a vector, forming a sequence of length (N), where N = HW / P. 2 H and W are the height and width of the image, respectively;

[0021] The third step of ViT inputs a sequence of length N into the transformer model. By utilizing the transformer's self-attention mechanism to process the pixel relationships in the image, the model is able to learn a global image representation, thereby extracting detailed information about the image and constructing the image's feature vector.

[0022] Text feature extraction module: Applying Natural Language Processing (NLP) technology, using the Transformer model, pre-training and learning the text data to extract the text feature vectors;

[0023] The first step of the text feature extraction module is to convert the input text into indices in the vocabulary;

[0024] The second step of the text feature extraction module is to find the embedding representation of the corresponding word through the index, and then input the embedding representation into the transformer model;

[0025] The third step in the text feature extraction module is that the transformer model outputs a feature vector containing text context information.

[0026] Multimodal fusion alignment module: It uses a comparative learning method of text-image pairs to train the alignment of text feature vectors with image feature vectors;

[0027] First, text and image feature vectors are generated by text feature extraction modules and image feature extraction modules, respectively. Then, the feature vector information is input into a multimodal fusion and alignment transformer model structure. Finally, by comparing and learning the image and text feature vectors, the multimodal fusion and alignment model can generate vector representations of text and images in the same vector space, that is, map text and image feature vectors to a unified space, providing feature vectors for subsequent cross-modal retrieval.

[0028] Text and images are input into the text feature extraction module and the image feature extraction module, respectively, to generate corresponding feature vectors. Then, the feature vectors are input into the multimodal fusion transformer model. Through cross-modal training, the training objective function is the KL divergence between the text feature vector and the image feature vector.

[0029] The formula for calculating KL divergence is: The higher the similarity between P and Q, the better D kl The smaller the value, the better.

[0030] Furthermore,

[0031] The similarity calculation and matching module calculates the similarity between the vector representation in the unified space obtained by the multimodal fusion alignment module and the vectors in the vector library, and returns the most relevant information to the user.

[0032] The cosine similarity method is used to calculate the similarity between feature vectors of different modalities; cosine_distance = 1 - (A·B) / (||A||*||B||), where A and B represent two vectors, · represents the dot product of the vectors (i.e., the sum of corresponding element-wise multiplication), and ||A|| and ||B|| represent the norms of the two vectors, i.e., the lengths of the vectors. By calculating the cosine distance, a value between 0 and 1 can be obtained. The closer the value is to 1, the more similar the two vectors are, and the closer the value is to 0, the less similar the two vectors are.

[0033] A data retrieval mechanism is established using the Faiss vector search engine. The Faiss vector retrieval library stores cross-modal features of text and images from existing knowledge bases. When a retrieval task arrives, the text or image is first converted into a cross-modal feature vector, and then the difference between this vector and the cross-modal features in the Faiss library is calculated to retrieve relevant image or text information for processing.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] (1) Higher retrieval accuracy: The feature vectors extracted by the deep learning model can more accurately express the semantic information of different modal information such as images and text, thereby improving the relevance and accuracy of the retrieval results.

[0036] (2) Faster response speed: By utilizing efficient vector search engines, such as Faiss, the system can achieve fast retrieval on large-scale datasets, significantly improving query response speed.

[0037] (3) Enhanced cross-modal retrieval capability: This invention allows users to perform cross-modal retrieval through a multimodal fusion mechanism, that is, to use images to query text information, or use text to query image information, or to query between other modalities, providing a more flexible retrieval method.

[0038] (4) Better scalability: The system design takes into account future data growth and technological development, and has good scalability and adaptability, which can continuously meet long-term business needs.

[0039] (5) Wider scope of application: This invention can be widely applied in many fields such as digital libraries, news archive retrieval, and online product search, and has broad market application prospects.

[0040] (6) More granular search results: The system can understand the user's query intent and return more accurate search results, improving the effectiveness of information retrieval.

[0041] In summary, the multimodal retrieval system of the present invention has significant advantages in improving retrieval performance, optimizing user experience, and expanding application scope, and is an important improvement over the prior art. Attached Figure Description

[0042] Figure 1 is a flowchart of the overall workflow of the present invention;

[0043] Figure 2 is a schematic diagram of the image feature extraction module in operation;

[0044] Figure 3 is a schematic diagram of the text feature extraction module in operation;

[0045] Figure 4 is a diagram of the model structure and a schematic diagram of the training objectives;

[0046] Figure 5 is a schematic diagram of the overall structure of the present invention;

[0047] Figure 6 is a schematic diagram of the index construction process of the present invention;

[0048] Figure 7 is a schematic diagram of the retrieval process of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0050] The core invention of this case is to map features from different modalities to a unified vector space through a Transform model, thereby aligning feature vectors from different modalities. Simply put, it can be described as ensuring the highest similarity between the text vector representation of "apple" and the vector representation of the apple image. This enables cross-modal retrieval of text to image or image to text, and can also be further extended to retrieval of other modalities.

[0051] The overall flowchart of this case is shown in Figure 1, and includes the following steps:

[0052] Step 1: Feature extraction from different modal inputs. The multimodal feature extraction module extracts feature vectors from the input information of different modalities.

[0053] Step 2: Different modal features are used to generate vector representations of a common feature space using the Transform model. The multimodal fusion alignment module transforms the feature vectors input in Step 1 into a unified vector space.

[0054] Step 3: Calculate the similarity between vector representations of different modalities, and retrieve relevant cross-modal information from the vector library according to the degree of similarity, returning it to the user. The similarity calculation and matching module calculates the similarity between the vector representations in the unified space obtained in Step 2 and the vectors in the vector library, obtaining the most relevant information and returning it to the user.

[0055] By following the three steps above, cross-modal retrieval of images to text and text to images can be achieved. The vectors in the multimodal vector library are intended to be uniformly vectorized from text and images of different modalities and stored in the vector library. When users need to retrieve relevant content, they can do so by following the three steps above.

[0056] The following is a detailed description of the key components of a multimodal retrieval system based on the similarity of image and text feature vectors:

[0057] The multimodal feature extraction module mainly includes an image feature extraction module and a text feature extraction module;

[0058] Image Feature Extraction Module: This module uses deep learning models such as ViT (Vision Transformer) to process the input image and extract its feature vectors. These feature vectors represent key visual information in the image, such as object shape, color, and texture. The first step of ViT is to segment the image into a series of patches, each containing (P^2) pixels, where P is the preset patch size. This converts the two-dimensional image data into a one-dimensional sequence, allowing the application of the Transformer model. The second step of ViT flattens each patch into a vector, forming a sequence of length (N), where N = HW / P. 2 H and W represent the height and width of the image, respectively. In the third step of ViT, a sequence of length N is input into the transformer model. By utilizing the transformer's self-attention mechanism to process pixel relationships in the image, the model can learn a global image representation. This allows it to extract detailed image information and construct the image's feature vector. The specific image feature extraction model results are shown in Figure 2.

[0059] The text feature extraction module applies Natural Language Processing (NLP) techniques, using the Transformer model for pre-training and learning on text data to extract feature vectors. These feature vectors capture key semantic information of the text content. The first step of the text feature extraction module is to convert the input text into word segments and then into indices in the vocabulary. The second step is to find the embedding representation of the corresponding word using the index, and then input the embedding representation into the Transformer model. The third step is for the Transformer model to output feature vectors containing text context information. The specific model structure is shown in Figure 3.

[0060] The text feature extraction module learns the language expression methods through unsupervised text training, and is ultimately used to extract text information to generate text vectors.

[0061] Multimodal fusion alignment module: This module designs a strategy to combine feature vectors from images and text to generate a vector representation in a common feature space, enabling cross-modal similarity retrieval. Multimodal fusion employs a comparative learning approach between text and image feature vectors for training the alignment of text and image feature vectors. First, text and image feature vectors are generated by text and image feature extraction modules, respectively. These feature vectors are then input into a multimodal fusion alignment transformer model. Finally, through comparative learning of image and text feature vectors, the multimodal fusion alignment model generates vector representations of text and images in the same vector space, mapping text and image feature vectors to a unified space, providing feature vectors for subsequent cross-modal retrieval. This module achieves high similarity between the feature vectors of the text "apple" and the feature vectors of the image "apple." This enables efficient and accurate vector retrieval across different modalities, helping users efficiently search for information across different modalities.

[0062] As shown in Figure 4, text and image are input into the text feature extraction module and image feature extraction module, respectively, generating corresponding feature vectors. These feature vectors are then input into the multimodal fusion transformer model. Through cross-modal training, the training objective function is the KL (Kullback-Leibler) divergence between the text and image feature vectors. The loss is calculated using the KL divergence to determine the difference between the two. Simply put, the goal is to minimize the difference between the text and image vectors; the smaller the difference, the more likely the text and image represent the same information. The KL divergence calculation formula is: As can be seen from the KL formula, the higher the similarity between P and Q, the smaller the value of Dkl.

[0063] Similarity Calculation and Matching Module: Develop efficient algorithms to calculate the similarity between query feature vectors and feature vectors stored in the database. This case uses cosine similarity as a metric to calculate the similarity between feature vectors of different modalities. cosine_distance = 1 - (A·B) / (||A||*||B||), where A and B represent two vectors, · represents the dot product of the vectors (i.e., summing corresponding element-wise), and ||A|| and ||B|| represent the norms (i.e., the lengths of the vectors). By calculating the cosine distance, a value between 0 and 1 can be obtained; the closer the value is to 1, the more similar the two vectors are, and the closer the value is to 0, the less similar the two vectors are.

[0064] Indexing and search optimization methods utilize the Faiss vector search engine to establish an efficient indexing mechanism, enabling rapid data retrieval. Optimized algorithms and techniques are employed to improve search efficiency on large-scale datasets. Through the open-source Faiss vector retrieval library, cross-modal features of text and images from existing knowledge bases are stored in the Faiss vector library. When a retrieval task arrives, the system first converts the text or image into cross-modal feature vectors, then calculates the difference between these vectors and the cross-modal features in the Faiss library, thereby retrieving relevant image or text information for processing.

[0065] Figure 4 is a complete system structure diagram illustrating the entire cross-modal calculation of the similarity of information across different modalities. Combined with the system flowcharts in the following 6 sections, cross-modal retrieval functions can be efficiently implemented.

[0066] The present invention aims to build a system that can accurately and efficiently process and retrieve multimodal information. By integrating image and text features, it provides a powerful cross-modal retrieval system.

[0067] Improve retrieval accuracy: By utilizing deep learning technology to extract high-dimensional feature vectors from images and text, the system can more accurately understand the semantic relationship between the query content and the stored data, thereby improving the relevance and accuracy of the retrieval results.

[0068] Optimize search response time: By utilizing efficient vector similarity search engines such as Faiss, the system can quickly locate relevant information in massive amounts of data, achieving millisecond-level response times.

[0069] Cross-modal retrieval: The system can not only handle single-modal retrieval tasks, but also perform cross-modal retrieval, that is, users can input images to search for related text information, or vice versa, which greatly improves the applicability and flexibility of the retrieval system.

[0070] In summary, the purpose of inventing a multimodal retrieval system based on image and text feature vector similarity is to provide users with an efficient, accurate, and user-friendly retrieval tool to meet the growing demand for multimodal information retrieval.

[0071] The above description is merely a preferred embodiment of the present invention and is used only to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A multi-modal retrieval system based on similarity of text and image feature vectors, characterized in that, Comprise: a multi-modal feature extraction module for extracting feature vectors of different modal input information; using a Transform model to generate vector representations of common feature spaces of different modal features; a multi-modal fusion alignment module for converting the feature vectors extracted by the multi-modal feature extraction module to a unified vector space; calculating the similarity between the vector representations of different modalities, and retrieving the cross-modal related information from the vector library according to the similarity degree and returning it to the user; a similarity calculation and matching module for calculating the similarity between the vector representations in the unified space obtained by the multi-modal fusion alignment module and the vectors in the vector library, and returning the most relevant information to the user.

2. The system of claim 1, wherein the multi-modal feature extraction module comprises an image feature extraction module and a text feature extraction module; wherein the image feature extraction module uses a ViT (Vision Transformer) deep learning model to process the input image to extract the feature vector of the image; the ViT first divides the image into a series of small patches, each containing P^2 pixel points, where P is a predetermined image block size; ViT second step divides the image into blocks, each block is flattened into a vector, forming a sequence of length (N), where N = HW / P 2 H and W are the height and width of the image, respectively; the ViT third step inputs a sequence of length N into the transformer model, which processes the pixel relationship in the image by using the transformer self-attention mechanism, which enables the model to learn the global image representation; thereby extracting detailed information of the image and constructing the feature vector of the image; the text feature extraction module applies natural language processing (NLP) technology and uses a Transformer model to pre-train and learn the text data to extract the feature vector of the text; the text feature extraction module first step converts the input text into an index in the word table through word segmentation; the text feature extraction module second step finds the embedding representation of the corresponding word through the index, and then inputs the embedding representation into the transformer model; the text feature extraction module third step, the transformer model outputs a feature vector containing text context information.

3. The system of claim 2, wherein the multi-modal fusion alignment module uses a text-image pair for contrastive learning to train the alignment of the text feature vector and the image feature vector; first, the text and image text generate corresponding feature vectors through the text feature extraction module and the image feature extraction module, then input the feature vector information into a multi-modal fusion alignment transformer model structure, finally through contrastive learning of image and text feature vectors, then the multi-modal fusion alignment model can generate vector representations of text and image in the same vector space, that is, map the text and image feature vectors to a unified space, providing feature vectors for subsequent cross-modal retrieval.

4. The system of claim 3, wherein Text and image are input into text feature extraction module and image feature extraction module respectively, corresponding feature vectors are generated, and then the feature vectors are input into the multi-modal fusion transformer model. Through cross-modal training, the training objective function is the KL divergence of the text feature vector and the image feature vector.

5. The system of claim 4, wherein, The KL divergence calculation formula is: The higher the similarity of P and Q, the smaller the value of D kl .

6. The system of claim 1, wherein, The similarity calculation and matching module: adopts the cosine similarity measurement method to calculate the similarity between different modal feature vectors; cosine_distance = 1 - (A · B) / (||A|| * ||B||) where, A and B represent two vectors respectively, · represents the dot product of vectors (i.e. multiply the corresponding elements and sum), ||A|| and ||B|| represent the norm of two vectors, i.e. the length of the vector; by calculating the cosine distance, a value between 0 and 1 can be obtained, the closer the value is to 1, the more similar the two vectors are, and the closer the value is to 0, the less similar the two vectors are.

7. The system of claim 1, wherein, An index mechanism is established by using the vector search engine faiss to realize data retrieval.

8. The system of claim 1, wherein, Through the faiss vector retrieval library, the cross-modal features in the existing knowledge base are stored in the faiss vector library. When a retrieval task comes, the text or image is first converted into a cross-modal feature vector, and then the difference between it and the cross-modal features in the faiss library is calculated, so as to retrieve the relevant picture or text information processing.

Images