Multi-modal image retrieval method and system
The multimodal image retrieval method addresses inefficiencies in traditional image retrieval by mapping image features to text space, using a text reversal network and encoder for accurate feature representation and fusion, improving accuracy and efficiency in large-scale data processing.
Patent Information
- Application Number
- CN202411189843.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional image retrieval methods face inefficiencies in processing large-scale data, insufficient processing capabilities for different semantic information, and accuracy issues in complex scenarios, especially when dealing with large-scale image and text data, which require significant computing resources and are sensitive to semantic gaps between image features and query text.
A multimodal image retrieval method that maps image features to text feature space using a multimodal algorithm, incorporating semantic information through a combination of input images and texts, and employs a text reversal network and encoder trained with knowledge distillation to generate pseudo-word tags and text features for accurate representation and fusion.
Enhances retrieval accuracy and efficiency by flexibly processing image-text relationships, reducing feature loss, and maintaining low resource consumption, suitable for resource-constrained devices and real-time applications.
Smart Images

Figure CN119088998A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image retrieval, and in particular to a multimodal image retrieval method and system. Background Art
[0002] Traditional image retrieval methods usually rely on directly comparing image features or searching through text descriptions, which are mainly divided into the following three types: First, by obtaining the features of images and texts, constructing and updating similarity nodes, transmitting similarity information, and using the attention mechanism to filter the nodes by semantic weights to determine the target similarity, cross-modal information retrieval can be achieved, and images can be retrieved through text or text can be retrieved through images. The defects of this technology are: insufficient processing efficiency for large-scale data, insufficient processing capabilities for different semantic information, and insufficient accuracy in complex scenarios, especially when processing large-scale image and text data, which requires a lot of computing resources. Second, use pre-trained encoder networks and decoder networks to extract the feature map features of the image to be retrieved and generate description texts, calculate the spatial attention weights of each pixel of the image feature map and the channel attention weights of each channel, generate feature vectors, and then use these feature vectors to compare with the feature vectors of the query text to achieve image retrieval. The defects of this technology are: in practical applications, if the number of images to be retrieved is very large, there will be delay problems. In addition, this method relies on the accuracy of feature extraction and attention weights. If the semantic gap between image features or description text and query text is large, it will affect the accuracy of retrieval results. Third, by using the preset feature extraction model to extract image feature vectors, and segmenting and unsupervised clustering these feature vectors to establish an index feature library, the retrieval efficiency and accuracy can be improved by quickly matching the feature vectors of each segment during retrieval. At the same time, the dimensionality reduction process reduces the computational complexity and improves the user experience. The drawbacks of this technology are: the effect of feature extraction and clustering is highly dependent on the quality of the preset feature extraction model. If the model is not fully trained or the data set is biased, the accuracy of the retrieval will be affected. This method is not sensitive enough to certain specific features of the image, resulting in incomplete retrieval results in some cases. Summary of the invention
[0003] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a multimodal image retrieval method, which can realize cross-modal image retrieval by mapping image features to text feature space based on a multimodal algorithm, allowing the system to more flexibly process and understand the relationship between image and text, thereby preventing some feature loss that may be caused by single image or single text retrieval, thereby affecting the accuracy of retrieval, and because semantic information is introduced by the method of performing image retrieval through a combination of input images and texts, it is possible to use semantic information to assist, rather than relying on certain specific features of the image, to a certain extent, so that the retrieval results are more comprehensive.
[0004] The present invention also proposes a multimodal image retrieval system for implementing the multimodal image retrieval method.
[0005] The present invention also proposes a computer device for implementing the multimodal image retrieval method.
[0006] The present invention also provides a computer-readable storage medium.
[0007] The multimodal image retrieval method according to the first aspect of the present invention comprises the following steps:
[0008] Train the text reversal network and text encoder;
[0009] Convert each image in the image library into a corresponding first text feature through the text inversion network and the text encoder;
[0010] Converting the image to be retrieved into a corresponding second text feature through the text inversion network and the text editor;
[0011] Convert the text to be retrieved into corresponding third text features through the text editor;
[0012] Embed and fuse the second text feature with the third text feature to obtain a fourth text feature;
[0013] The similarity between the fourth text feature and each of the first text features is calculated, and according to the similarity, a matching image in the image library that matches the image to be retrieved and the text to be retrieved is determined.
[0014] According to some embodiments of the present invention, converting each image in the image library into a corresponding first text feature through the text inversion network and the text encoder comprises:
[0015] Convert each image in the image library into a corresponding first pseudo-word tag through the text inversion network;
[0016] converting the first pseudo-word token into a corresponding first text feature by the text editor,
[0017] The converting the image to be retrieved into the corresponding second text feature by the text inversion network and the text editor comprises:
[0018] Converting the image to be retrieved into a corresponding second pseudo-word tag through the text inversion network;
[0019] The second pseudo-word token is converted into a corresponding second text feature by the text editor.
[0020] According to some embodiments of the present invention, the step of training the text reversal network includes:
[0021] Acquire training data, wherein the training data includes a plurality of image data and text data corresponding to the plurality of image data;
[0022] The training data is passed to the text reversal network through knowledge distillation, so that the text reversal network can generate corresponding pseudo-word tags in a single forward propagation.
[0023] According to some embodiments of the present invention, the step of embedding and fusing the second text feature with the third text feature comprises:
[0024] The second text feature and the third text feature are embedded and fused by using vector concatenation, weighted summation or attention mechanism.
[0025] According to some embodiments of the present invention, before calculating the similarity between the fourth text feature and each of the first text features, the following steps are also included:
[0026] The fourth text feature is normalized, and the normalization adopts L2 regularization or batch normalization method to ensure that the fourth text feature has a uniform scale in the embedding space.
[0027] According to some embodiments of the present invention, the text encoder is trained based on a BERT or Transformer pre-trained model, so that the text encoder can generate text features that accurately represent the visual information of the original image.
[0028] According to some embodiments of the present invention, the step of calculating the similarity between the fourth text feature and each of the first text features includes:
[0029] The similarity between the fourth text feature and each of the first text features is calculated by using a similarity measurement method and the FAISS library as a feature similarity retrieval and clustering tool.
[0030] A multimodal image retrieval system according to a second aspect of the present invention includes:
[0031] Training module, used to train text reversal network and text encoder;
[0032] A feature generation module, configured to convert each image in the image library into a corresponding first pseudo-word tag through the text inversion network, and then convert the first pseudo-word tag into a corresponding first text feature through the text encoder;
[0033] an embedding module, configured to convert the image to be retrieved into a corresponding second pseudo-word tag through the text inversion network, and then convert the second pseudo-word tag into a corresponding second text feature through the text encoder; and to convert the text to be retrieved into a corresponding third text feature through the text editor, and then embed and fuse the second text with the third text feature to obtain a fourth text feature;
[0034] The retrieval module is used to calculate the similarity between the fourth text feature and each of the first text features, and determine, based on the similarity, a matching image in the image library that matches the image to be retrieved.
[0035] A computer device according to an embodiment of a third aspect of the present invention includes:
[0036] one or more processors;
[0037] Memory;
[0038] One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the multimodal image retrieval method as described in any one of the embodiments of the first aspect are implemented.
[0039] According to the computer-readable storage medium of the fourth aspect embodiment of the present invention, a computer program is stored thereon, and when the computer program is executed by a processor, the steps of the multimodal image retrieval method as described in any one of the embodiments of the first aspect are implemented.
[0040] The multimodal image retrieval method and system according to the embodiments of the present invention have at least the following beneficial effects: based on the multimodal algorithm, cross-modal image retrieval is achieved by mapping image features to text feature space, which allows the system to more flexibly process and understand the relationship between image and text, thereby avoiding feature loss that may be caused by mutual retrieval between image and text or retrieval of images only through text, thereby affecting the accuracy of image retrieval; and image retrieval is performed by inputting a combination of image and text, introducing semantic information, which to a certain extent does not rely on certain specific features of the image, but uses semantic information as an aid, making the retrieval results more comprehensive.
[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0043] Figure 1 A schematic diagram of a flow chart of a multimodal image retrieval method according to an embodiment of the present invention;
[0044] Figure 2 for Figure 1 The multimodal image retrieval method shown here trains a text inversion network and a text encoder.
[0045] Figure 3 for Figure 1 A detailed flow chart of the steps of generating the first text feature and the second text feature is shown;
[0046] Figure 4 for Figure 1 A detailed flow chart of the steps of obtaining the first text feature, the second text feature and the third text feature, embedding and fusing the second text feature and the third text feature to obtain the fourth text feature, and returning a similar image is shown;
[0047] Figure 5 A module connection diagram of a multimodal image retrieval system according to an embodiment of the present invention.
[0048] Training module 100, text reversal network 110, text encoder 120,
[0049] Feature generation module 200, image library image 210,
[0050] Embedding module 300, image to be retrieved 310, text to be retrieved 320,
[0051] Retrieval module 400. DETAILED DESCRIPTION
[0052] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.
[0053] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.
[0054] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0055] The following is combined with Figure 1-5 , describe in detail the multimodal image retrieval method and system of the embodiments of the present invention.
[0056] Reference Figure 1 and Figure 4 , the present invention proposes a multimodal image retrieval method, comprising:
[0057] Step S100: training the text reversal network 110 and the text encoder 120;
[0058] Step S200: converting each image 210 in the image library into a corresponding first text feature through the text inversion network 110 and the text encoder 120;
[0059] Step S300: converting the image to be retrieved 310 into a corresponding second text feature through the text inversion network 110 and the text editor 120;
[0060] Step S400: converting the to-be-searched text 320 into a corresponding third text feature through the text editor 120;
[0061] Step S500: embed and fuse the second text feature with the third text feature to obtain a fourth text feature;
[0062] Step S600: Calculate the similarity between the fourth text feature and each first text feature, and determine the matching image in the image library image 210 that matches the image to be retrieved 310 and the text to be retrieved 320 according to the similarity.
[0063] Specifically, in this embodiment, when training the text reversal network 110, the image is iteratively optimized through the text reversal technology to generate pseudo-word tags, and a lightweight text reversal network 110 is trained so that it can map the visual features of the input image to the pseudo-word tags in the embedding space; when training the text encoder 120, based on the BERT or Transformer model, it is enabled to convert the pseudo-word tags into text features, and the text feature refers to the vector representation of the text data in the embedding space. The trained text reversal network 110 and text encoder 120 together ensure that the generated text features can accurately represent the visual information of the original input image. Figure 3 After the text reversal network 110 and the text encoder 120 are trained, each image library image 210 is converted into a corresponding first text feature through the text reversal network 110 and the text encoder 120; the image to be retrieved 310 is then converted into a second text feature through the text reversal network 110 and the text encoder 120, and the text to be retrieved 320 is converted into a third text feature through the text encoder 120; then, a feature vector fusion method such as vector concatenation, weighted summation or attention mechanism is used to embed and fuse the second text feature with the third text feature in the embedding space to obtain a fourth text feature; finally, the similarity between the fourth text feature and each first text feature is calculated and compared, and the retrieval results are sorted according to the similarity, and the matching image in the image library image 210 that best matches the image to be retrieved 310 and the text to be retrieved 320 is returned, thereby completing multimodal image retrieval. By using the text reversal technology, the image visual features of the image to be retrieved 310 and the image library image 210 are converted into pseudo-word tags, so that the image and text can be represented and compared in the same embedding space, thereby achieving effective fusion of cross-modal data, greatly improving the accuracy and efficiency of retrieval. By training a lightweight text reversal network 110, image features can be accurately converted into text features while maintaining efficient computing and low resource consumption. This method ensures the efficiency and real-time performance of the text reversal network 110, making the method suitable for resource-constrained devices and application scenarios with high real-time requirements, such as image search on mobile devices and online real-time image retrieval.
[0064] Reference Figure 3 , further, in some embodiments of the present invention, step S200: converting each image 210 in the image library into a corresponding first text feature through the text inversion network 110 and the text encoder 120 includes:
[0065] S210: converting each image 210 in the image library into a corresponding first pseudo-word tag through the text inversion network 110;
[0066] S220: Convert the first pseudo-word token into a corresponding first text feature through the text encoder 120.
[0067] Step S300: converting the image to be retrieved 310 into a corresponding second text feature through the text inversion network 110 and the text encoder 120, including:
[0068] S310: Convert the to-be-retrieved image 310 into a corresponding second pseudo-word tag through the text inversion network 110;
[0069] S320: Convert the second pseudo-word token into a corresponding second text feature through the text encoder 120.
[0070] Specifically, first, the trained text inversion network 110 is used to invert each image 210 in the image library to be retrieved, and a corresponding text description or pseudo-word tag, i.e., a first pseudo-word tag, is generated. These first pseudo-word tags are converted into corresponding text feature vectors, i.e., first text features, through the text encoder 120. All first text features together constitute a text feature library corresponding to the original image library, and the feature library contains the text feature representation of each image 210 in the image library. Subsequently, the trained text inversion network 110 is used again to process the image 310 to be retrieved through the text inversion network 110, and the visual features of the image 310 to be retrieved are converted into corresponding text descriptions or pseudo-word tags, i.e., second pseudo-word tags. Subsequently, the generated second pseudo-word tags are converted into corresponding text feature vectors, i.e., second text features, through the trained text encoder 120. By converting all image features into text feature vectors, the subsequent fusion of feature vectors is facilitated, thereby realizing the fusion of cross-modal data.
[0071] Reference Figure 2 Further, in some embodiments of the present invention, in step S100: training the text reversal network 110 and the text encoder 120, the step of training the text reversal network 110 includes:
[0072] Step S110: obtaining training data;
[0073] Step S120: passing the training data to the text reversal network 110 through knowledge distillation, so that the text reversal network 110 can generate corresponding pseudo-word tags in a single forward propagation.
[0074] Specifically, when training the text reversal network 110, the text reversal technology is used, and the text reversal technology generally involves the following steps: 1. Collecting samples and creating text descriptions: The user needs to prepare some images with specific features and provide corresponding text descriptions for these images, that is, step S110: obtaining training data, wherein the training data includes a number of image data and text data corresponding to the number of image data; 2. Using these text descriptions and corresponding sample images to train an embedding model, the model will learn how to map text descriptions to image features, that is, in step S120: passing the training data to the text reversal network 110 through knowledge distillation, so that the text reversal network 110 can generate corresponding pseudo-word tags in a single forward propagation. The knowledge distillation method used in step S120 is a technology that transfers the knowledge of a large and complex model (teacher model) to a small and simple model (student model). The use of this technology can bring the following technical effects: 1. Improving the accuracy of the training model (i.e., the text reversal network 110): By imitating the output and intermediate layer features of the teacher model (i.e., training data), the student model (i.e., the text reversal network 110) can significantly improve the performance without changing the structure; 2. Reducing the model (i.e., the text reversal network 110) delay and compressing network parameters: The knowledge distillation method can effectively reduce the computational complexity and storage requirements of the model (i.e., the text reversal network 110), so that the model (i.e., the text reversal network 110) can also run efficiently on resource-constrained hardware, thereby Achieve effects that can be applied on edge devices and mobile devices; 3. Reduce the amount of annotations: Knowledge distillation can reduce the need for a large amount of annotated data by migrating existing knowledge, thereby reducing the training cost of the model (i.e., the text reversal network 110); 4. Improve reasoning efficiency: Due to the lightweight nature of the student model (i.e., the text reversal network 110), it can complete tasks at a faster speed, especially in application scenarios that require real-time processing, thereby improving the real-time performance of the text reversal network 110 in generating pseudo-word tags; 5. Domain transfer capability: Knowledge distillation is not limited to the transmission of label information, but also includes multi-dimensional knowledge transfer such as features and intermediate layer representations, so that the student model (i.e., the text reversal network 110) can better adapt to new tasks or data sets.
[0075] Reference Figure 1 Further, in some embodiments of the present invention, step S500: embedding and fusing the second text feature with the third text feature to obtain a fourth text feature includes:
[0076] The second text feature is embedded and fused with the third text feature by using vector concatenation, weighted summation or attention mechanism.
[0077] Specifically, when fusing text features, the second text features and the third text features from different feature extraction methods are combined to obtain a more comprehensive and rich text representation method. This fusion can be achieved in a variety of ways, including feature concatenation, cross-modal attention mechanism, conditional batch normalization, and weighted summation. By fusing the embedding of the second text feature and the third text feature, the image to be retrieved 310 and the text to be retrieved 320 can be represented and compared in the same embedding space, thereby achieving effective fusion of cross-modal data and greatly improving the accuracy and efficiency of subsequent retrieval.
[0078] Reference Figure 1 Further, in some embodiments of the present invention, before step S600: calculating the similarity between the fourth text feature and each first text feature, the following steps are also included:
[0079] The fourth text feature is normalized, and the normalization process adopts L2 regularization or batch normalization method to ensure that the fourth text feature has a uniform scale in the embedding space.
[0080] Specifically, when the fourth text feature is normalized, an L2 regularization method or a batch normalization method can be used to scale each dimension of all feature vectors in the fourth text feature to a fixed range, such as a specific norm range or a unit length, so as to convert them into unit vectors, that is, each feature vector is divided by its length, and the length can be measured using an L2 norm (Euclidean distance) or an L1 norm (Euclidean distance). By normalizing the fourth text feature, the dimensional differences of the feature vectors in the fourth text feature can be eliminated, so that all feature vectors can be compared and calculated on the same scale.
[0081] Reference Figure 1 , Figure 2 and Figure 4, Further, in some embodiments of the present invention, in step S100: training the text inversion network 110 and the text encoder 120, the text encoder 120 is trained based on a BERT or Transformer pre-trained model, so that the text encoder 120 can generate text features that accurately represent the visual information of the original image. Specifically, the purpose of training the text encoder 120 is to convert text data into a numerical representation that can be understood and processed by the machine learning model, namely, a feature vector, so as to facilitate subsequent retrieval in the embedding space; the specific steps include: 1. First, it is necessary to prepare training data sets, which contain a large amount of text data for training the model to capture language features; 2. Configure the appropriate hardware and software environment, including installing the necessary deep learning framework (such as TensorFlow or PyTorch) and related dependent libraries; 3. Build the encoder part of the Transformer model, which involves defining various components of the model, including self-attention layer, multi-head attention mechanism, feedforward fully connected layer, normalization layer and residual connection, etc.; 4. Select appropriate hyperparameters, including the selection of parameters such as learning rate, batch size, and number of training rounds; 5. Start training, including: forward propagating the input data through the pre-trained encoder, calculating the representation vector of each word or word, calculating the loss of the model according to the target label, and then using the back propagation of the error signal to update the weight of the model to minimize the loss, and finally select and use an appropriate optimizer (such as Adam) to adjust the model parameters. 6. Adopt some regularization strategies to prevent overfitting, such as Droput, L2 regularization, etc.; 7. Regularly evaluate the performance of the model and adjust the model according to the evaluation results, such as adjusting hyperparameters or adding more training data, so as to complete the training of the text encoder 120.
[0082] Reference Figure 1 Further, in some embodiments of the present invention, in step S600, the step of calculating the similarity between the fourth text feature and each first text feature includes:
[0083] By using the similarity measurement method, the similarity between the fourth text feature and each first text feature is calculated using the FAISS library as a feature similarity retrieval and clustering tool.
[0084] Specifically, FAISS (Facebook AI Similarity Search) is an efficient similarity search and clustering tool, especially suitable for processing large-scale vector collections. When using the FAISS library for feature retrieval, first use the FAISS library to build a vector index. FAISS provides a variety of index structures, such as IndexFlatL2, as well as IVF and PQ methods, which can significantly improve the speed and efficiency of similarity search. After the index is built, perform similarity retrieval of feature vectors: given a query vector, FAISS can quickly find the most similar feature vector. This process does not require comparing the query vector with each vector in the database one by one, but speeds up the search through the index, thereby improving the retrieval speed and retrieval accuracy.
[0085] Reference Figure 3 and Figure 4 The present invention also proposes a multimodal image retrieval system, comprising:
[0086] A training module 100, for training a text reversal network 110 and a text encoder 120;
[0087] A feature generation module 200, configured to convert each image 210 in the image library into a corresponding first pseudo-word tag through the text inversion network 110, and then convert the first pseudo-word tag into a corresponding first text feature through the text encoder 120;
[0088] The embedding module 300 is used to convert the image to be retrieved 310 into a corresponding second pseudo-word tag through the text inversion network 110, and then convert the second pseudo-word tag into a corresponding second text feature through the text encoder 120; and to convert the text to be retrieved 320 into a corresponding third text feature through the text editor 120, and then embed and fuse the second text feature with the third text feature to obtain a fourth text feature;
[0089] The retrieval module 400 is used to calculate the similarity between the fourth text feature and each first text feature, and determine the matching image in the image library image 210 that matches the image to be retrieved 310 and the text to be retrieved 320 according to the similarity.
[0090] Specifically, through systematic modular design and process optimization, the fusion and retrieval of image features and text features in the embedding space are made more efficient, especially the functional integration of the feature generation module 200 and the embedding module 300, which significantly improves the accuracy and timeliness of image retrieval and solves the problems of complex calculation and poor real-time performance in traditional image retrieval methods.
[0091] The present invention also proposes a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the multimodal image retrieval method as described above are implemented.
[0092] The present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the multimodal image retrieval method as described above are implemented.
[0093] It will be appreciated by those skilled in the art that all or part of the steps for implementing the above-mentioned embodiments can be accomplished by hardware, or by a program to instruct the relevant hardware to accomplish, and the program can be stored in a computer-readable storage medium. In the context of the present invention, the computer-readable medium can be considered to be tangible and non-temporary. Non-limiting examples of non-temporary tangible computer-readable media include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits or dynamic random access memory circuits), magnetic storage media (e.g., analog or digital tapes or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs), etc. The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
[0094] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present invention. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.
[0095] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the technical field without departing from the purpose of the present invention.
Claims
1. A multimodal image retrieval method, It is characterized in that The following steps are involved: Train the text reversal network and text encoder; Convert each image in the image library into a corresponding first text feature through the text inversion network and the text encoder; Converting the image to be retrieved into a corresponding second text feature through the text inversion network and the text editor; Convert the text to be retrieved into corresponding third text features through the text editor; Embed and fuse the second text feature with the third text feature to obtain a fourth text feature; The similarity between the fourth text feature and each of the first text features is calculated, and according to the similarity, a matching image in the image library that matches the image to be retrieved and the text to be retrieved is determined.
2. The method according to claim 1, It is characterized in that The converting each image in the image library into a corresponding first text feature through the text inversion network and the text encoder comprises: Convert each image in the image library into a corresponding first pseudo-word tag through the text inversion network; converting the first pseudo-word token into a corresponding first text feature by the text editor, The converting the image to be retrieved into the corresponding second text feature by the text inversion network and the text editor comprises: Converting the image to be retrieved into a corresponding second pseudo-word tag through the text inversion network; The second pseudo-word token is converted into a corresponding second text feature by the text editor.
3. The method according to claim 1, It is characterized in that The step of training the text reversal network comprises: Get training data; The training data is passed to the text reversal network through knowledge distillation, so that the text reversal network can generate corresponding pseudo-word tags in a single forward propagation.
4. The method according to claim 1, It is characterized in that The step of embedding and fusing the second text feature with the third text feature comprises: The second text feature and the third text feature are embedded and fused by using vector concatenation, weighted summation or attention mechanism.
5. The method according to claim 1, It is characterized in that Before calculating the similarity between the fourth text feature and each of the first text features, the following steps are also included: The fourth text feature is normalized, and the normalization adopts L2 regularization or batch normalization method to ensure that the fourth text feature has a uniform scale in the embedding space.
6. The method according to claim 1, It is characterized in that The text encoder is trained based on a BERT or Transformer pre-trained model, so that the text encoder can generate text features that accurately represent the visual information of the original image.
7. The method according to claim 1, It is characterized in that The step of calculating the similarity between the fourth text feature and each of the first text features comprises: The similarity between the fourth text feature and each of the first text features is calculated by using a similarity measurement method and the FAISS library as a feature similarity retrieval and clustering tool.
8. A multimodal image retrieval system, It is characterized in that include: Training module, used to train text reversal network and text encoder; A feature generation module, configured to convert each image in the image library into a corresponding first pseudo-word tag through the text inversion network, and then convert the first pseudo-word tag into a corresponding first text feature through the text encoder; An embedding module, configured to convert the image to be retrieved into a corresponding second pseudo-word tag through the text inversion network, and then convert the second pseudo-word tag into a corresponding second text feature through the text encoder; and used to convert the to-be-searched text into a corresponding third text feature through the text editor, and then embed and fuse the second text with the third text feature to obtain a fourth text feature; The retrieval module is used to calculate the similarity between the fourth text feature and each of the first text features, and determine, based on the similarity, a matching image in the image library that matches the image to be retrieved and the text to be retrieved.
9. A computer device, It is characterized in that include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the multimodal image retrieval method as described in any one of claims 1-7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the multimodal image retrieval method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Image and text mixed retrieval method and system based on image embedding and data enhancement
CN118152518A
Multi-modal combined image retrieval method and system based on fine-grained text inversion
CN118193768A
