A cross-modal retrieval method for image-text

Abstract

The present invention discloses a cross-modal retrieval method for image-text, including obtaining an image-text pair data set and constructing a cross-modal image-text retrieval initial model; processing the image-text pair data set to obtain a feature vector; obtaining a unified dimensional feature vector of the feature vector in a common space, performing label classification and calculating the label classification loss; performing weighted sampling based on multiple negative samples on the image-text pair data set and calculating the weighted contrast loss of multiple negative samples; optimizing the cross-modal image-text retrieval initial model through an optimizer to obtain a cross-modal image-text retrieval model; and using the cross-modal image-text retrieval model to perform actual image-text cross-modal retrieval. The present invention pre-constructs a cross-modal image-text retrieval model, projects the image / text feature vector into a unified common space, introduces multiple negative sample sampling and weighted learning to perform model training, and uses the trained model to perform cross-modal retrieval; therefore, the method of the present invention has high retrieval accuracy, good reliability and fast retrieval speed.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a cross-modal retrieval method for image-text. Background Art

[0002] With the development of economy and technology, search technology has been widely used in people's production and life, bringing endless convenience to people's production and life.

[0003] A large amount of data is generated anytime and anywhere on the Internet, including multimedia information such as text, images, videos, and audio. This information is rich in content and diverse in form, which makes it urgent to solve the problem of how to obtain the information that is really needed from the massive data. At the same time, the large-scale growth of multimedia data urgently requires more effective technologies to achieve effective retrieval of multimedia data. Traditional retrieval only stays in a single mode, that is, input text to retrieve text, input image to retrieve image. In today's era of big data, traditional single-modal retrieval can no longer meet people's new needs for information retrieval.

[0004] Cross-modal retrieval means that the query data and retrieval results come from different modalities, such as finding the product image that best matches the text, or finding the most suitable text description for an image. Cross-module retrieval breaks away from the limitations of traditional single-modal retrieval and realizes cross-modal retrieval for different modalities. Its practical applicability is very strong. Therefore, cross-modal information retrieval has broad application prospects and important research significance.

[0005] The core of existing cross-modal retrieval technology is to use linear projection or deep learning technology to learn a common subspace, so that data of different modalities can be measured and learned in this common subspace for semantic similarity, and then return retrieval results similar to the query data through sorting. Nikhil et al. proposed an image-text retrieval method using canonical correlation analysis, obtained the feature projection vector in the subspace through linear transformation, and then maximized the correlation between the two modalities. Ding et al. proposed collaborative matrix decomposition to implement a cross-modal retrieval method, which finds the common semantics of data of different modalities through collaborative matrix decomposition and then projects it into a common space. These methods adopt a variety of methods for feature projection, but their sampling methods are very simple, all based on few-sample sampling and loss measurement, and the optimization gradient lacks flexibility. Summary of the invention

[0006] The object of the present invention is to provide a cross-modal retrieval method for image-text with high retrieval accuracy, good reliability and fast retrieval speed.

[0007] The cross-modal image-text retrieval method provided by the present invention comprises the following steps:

[0008] S1. Obtain a dataset of image-text pairs and build an initial model for cross-modal image-text retrieval;

[0009] S2. Processing the image and text data set obtained in step S1 to obtain a feature vector;

[0010] S3. The feature vector obtained in step S2 is projected through a projection function to obtain a feature vector of uniform dimension in a common space, perform label classification, and calculate the label classification loss;

[0011] S4. Perform weighted sampling based on multiple negative samples on the image-text pair dataset and calculate the weighted contrast loss of multiple negative samples;

[0012] S5. According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, the constructed cross-modal image-text retrieval initial model is optimized by an optimizer to obtain a cross-modal image-text retrieval model;

[0013] S6. Use the cross-modal image-text retrieval model obtained in step S5 to perform actual image-text cross-modal retrieval.

[0014] The step S1 of obtaining a data set of image-text pairs and constructing an initial model for cross-modal image-text retrieval specifically includes the following steps:

[0015] A. Obtaining a picture-text pair dataset: The picture-text pair dataset includes an image dataset and a text dataset;

[0016] B. The constructed initial model of cross-modal image-text retrieval includes: extracting image feature vector sets from image datasets through convolutional neural networks; extracting text feature vector sets from text datasets through bag-of-words models; then projecting the image feature vector sets and text feature vector sets into a common space through a projection function; and finally performing similarity measurement and label classification in the common space.

[0017] Step S2 of processing the image-text pair data set obtained in step S1 to obtain a feature vector specifically includes the following steps:

[0018] The image dataset in the image-text pair dataset obtained in step S1 is subjected to a convolutional neural network to extract image features, thereby obtaining an image feature vector set; the text dataset in the image-text pair dataset obtained in step S1 is subjected to a bag-of-words model to extract key semantic words, thereby obtaining a text feature vector set.

[0019] Step S3, in which the feature vector obtained in step S2 is projected through a function to obtain a feature vector of uniform dimension in a common space, performs label classification and calculates the label classification loss, specifically includes the following steps:

[0020] a. Pass the image feature vectors in the image feature vector set obtained in step S2 through a three-layer fully connected layer network;

[0021] b. Pass the text feature vectors in the text feature vector set obtained in step S2 through a three-layer fully connected layer network;

[0022] c. Through the processing of steps a and b, the image feature vector and the text feature vector are located in the same real-valued space, and then parameters are shared through the same fully connected layer, and finally projected into the same low-dimensional public space for label classification;

[0023] d. Use the following formula to calculate the label classification loss L1:

[0024]

[0025] Where n is the number of training samples; y i is the label for each training instance; p i (u i ) is the probability distribution of the generated image; p i (v i ) is the probability distribution of the generated text.

[0026] The step S4 described in which the image-text pair data set is subjected to weighted sampling based on multiple negative samples and the weighted contrast loss of multiple negative samples is calculated, specifically comprises the following steps:

[0027] If the anchor point is image modality data, a positive sample and several negative samples of text modality data are collected, the distance between the anchor point and the positive sample, and the distance between the anchor point and several negative samples are calculated, and the comparison weight values ​​of the positive sample and the negative sample are adjusted so that the distance between the anchor point and the positive sample in the common space is as close as possible and the distance between the anchor point and each negative sample in the common space is as far as possible;

[0028] If the anchor point is text modal data, a positive sample and several negative samples of the image modal data are collected, the distance between the anchor point and the positive sample, and the distance between the anchor point and several negative samples are calculated, and the comparison weight values ​​of the positive sample and the negative sample are adjusted so that the distance between the anchor point and the positive sample in the common space is as close as possible and the distance between the anchor point and each negative sample in the common space is as far as possible;

[0029] The following formula is used to calculate the weighted contrast loss L2 of multiple negative samples:

[0030]

[0031] Where w u is the weight when the anchor point is image modality data; f ()is the similarity measurement function; u is the image modality anchor point; v+ is the positive sample text similar to the image modality anchor point u; N1 is the number of sampled negative samples that are not similar to the image modality anchor point u; v k is the kth negative sample sampled and dissimilar to the image modality anchor point u; w v is the weight when the anchor point is text modality data; v is the text modality anchor point; u+ is the positive sample image similar to the text modality anchor point v; N2 is the number of sampled negative samples that are not similar to the text modality anchor point v; u k is the kth negative sample sampled that is dissimilar to the text modality anchor v.

[0032] The calculation of the distance between the anchor point and the positive sample, and between the anchor point and a number of negative samples, specifically calculates the cosine distance between the anchor point and the positive sample, and between the anchor point and a number of negative samples.

[0033] According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, step S5 optimizes the constructed cross-modal image-text retrieval initial model through an optimizer to obtain a cross-modal image-text retrieval model, which specifically includes the following steps:

[0034] According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, the Adam optimizer is used to update and continuously optimize the parameters in the initial model of cross-modal image and text retrieval, so as to obtain the final cross-modal image and text retrieval model.

[0035] Step S6 uses the cross-modal image-text retrieval model obtained in step S5 to perform actual image-text cross-modal retrieval, which specifically includes the following steps:

[0036] When the retrieval object is text modal data, the distance between the text modal data and each data in the image retrieval library is calculated, and according to the obtained distance, the best several image data are selected as the final retrieval result for feedback, thereby completing the actual image-text cross-modal retrieval;

[0037] When the retrieval object is image modal data, the distance between the image modal data and each data in the text retrieval library is calculated, and according to the obtained distance, the optimal number of text data is selected as the final retrieval result for feedback, thereby completing the actual image-text cross-modal retrieval.

[0038] The cross-modal retrieval method for image-text provided by the present invention pre-builds a cross-modal image-text retrieval model, inputs image data and text data, extracts feature vectors through an image processing network and a text processing network, and extracts effective information from the data; uses a projection function to project the image feature vector and the text feature vector to a unified public space to improve the matching speed; introduces multi-negative sample sampling and weighted learning to shorten the distance between positive sample pairs in the public space and lengthen the distance between negative sample pairs, with high semantic differentiation; uses a trained model to perform cross-modal retrieval, measures and sorts the data based on cosine distance, and returns the top-ranked data as the retrieval result, with high retrieval precision and strong robustness; therefore, the retrieval method of the present invention has high retrieval accuracy, good reliability and fast retrieval speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0040] like Figure 1 The method flow chart of the method of the present invention is shown as follows: The cross-modal retrieval method for image-text provided by the present invention comprises the following steps:

[0041] S1. Obtain a dataset of image-text pairs and build an initial model for cross-modal image-text retrieval; specifically, the following steps are included:

[0042] A. Obtaining a picture-text pair dataset: The picture-text pair dataset includes an image dataset and a text dataset;

[0043] B. The constructed cross-modal image-text retrieval initial model includes: extracting image feature vectors from image datasets through convolutional neural networks; extracting text feature vectors from text datasets through bag-of-words models; then projecting image feature vectors and text feature vectors to a common space through a projection function; and finally performing similarity measurement and label classification in the common space.

[0044] S2. Processing the image-text pair data set obtained in step S1 to obtain a feature vector; specifically comprising the following steps:

[0045] The image dataset in the image-text pair dataset obtained in step S1 is subjected to a convolutional neural network to extract image features, thereby obtaining an image feature vector set; the text dataset in the image-text pair dataset obtained in step S1 is subjected to a bag-of-words model to extract key semantic words, thereby obtaining a text feature vector set;

[0046] S3. The feature vector obtained in step S2 is projected through a projection function to obtain a feature vector of uniform dimension in the public space, perform label classification and calculate the label classification loss; specifically, the steps include:

[0047] a. Pass the image feature vectors in the image feature vector set obtained in step S2 through a three-layer fully connected layer network;

[0048] b. Pass the text feature vectors in the text feature vector set obtained in step S2 through a three-layer fully connected layer network;

[0049] c. Through the processing of steps a and b, the image feature vector and the text feature vector are located in the same real-valued space, and then parameters are shared through the same fully connected layer, and finally projected into the same low-dimensional public space for label classification;

[0050] d. Use the following formula to calculate the label classification loss L1:

[0051]

[0052] Where n is the number of training samples; y i is the label for each training instance; p i (u i ) is the probability distribution of the generated image; p i (v i ) is the generated text probability distribution;

[0053] S4. Perform weighted sampling based on multiple negative samples on the image-text pair dataset and calculate the weighted contrast loss of multiple negative samples; specifically, the steps include:

[0054] If the anchor point is image modality data, a positive sample and several negative samples of text modality data are collected, the distances (preferably cosine distances) between the anchor point and the positive sample, and between the anchor point and several negative samples are calculated, and the comparison weights of the positive sample and the negative sample are adjusted so that the distance between the anchor point and the positive sample in the common space is as close as possible and the distance between the anchor point and each negative sample in the common space is as far as possible;

[0055] If the anchor point is text modal data, a positive sample and several negative samples of the image modal data are collected, the distances (preferably cosine distances) between the anchor point and the positive sample, and between the anchor point and several negative samples are calculated, and the comparison weights of the positive sample and the negative sample are adjusted so that the distance between the anchor point and the positive sample in the common space is as close as possible and the distance between the anchor point and each negative sample in the common space is as far as possible;

[0056] The following formula is used to calculate the weighted contrast loss L2 of multiple negative samples:

[0057]

[0058] Where w u is the weight when the anchor point is image modality data; f ()is the similarity measurement function; u is the image modality anchor point; v+ is the positive sample text similar to the image modality anchor point u; N1 is the number of sampled negative samples that are not similar to the image modality anchor point u; v k is the kth negative sample sampled and dissimilar to the image modality anchor point u; w v is the weight when the anchor point is text modality data; v is the text modality anchor point; u+ is the positive sample image similar to the text modality anchor point v; N2 is the number of sampled negative samples that are not similar to the text modality anchor point v; u k is the kth negative sample sampled and dissimilar to the text modality anchor v;

[0059] S5. According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, the constructed cross-modal image-text retrieval initial model is optimized by an optimizer to obtain a cross-modal image-text retrieval model; specifically, the steps include:

[0060] According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, the Adam optimizer is used to update and continuously optimize the parameters in the initial model of cross-modal image and text retrieval, thereby obtaining the final cross-modal image and text retrieval model;

[0061] S6. Using the cross-modal image-text retrieval model obtained in step S5, actual image-text cross-modal retrieval is performed; specifically comprising the following steps:

[0062] When the retrieval object is text modal data, the distance (preferably cosine distance) between the text modal data and each data in the image retrieval library is calculated, and according to the obtained distance (preferably cosine distance), the best several image data are selected as the final retrieval result for feedback, thereby completing the actual image-text cross-modal retrieval;

[0063] When the retrieval object is image modal data, the distance (preferably cosine distance) between the image modal data and each data in the text retrieval library is calculated, and according to the obtained distance (preferably cosine distance), the best several text data are selected as the final retrieval result for feedback, thereby completing the actual image-text cross-modal retrieval.

Claims

1. A cross-modal retrieval method for image-text, comprising the following steps: S1. Obtain a dataset of image-text pairs and build an initial model for cross-modal image-text retrieval; S2. Processing the image and text data set obtained in step S1 to obtain a feature vector; S3. The feature vector obtained in step S2 is projected through a projection function to obtain a feature vector of uniform dimension in a common space, perform label classification, and calculate the label classification loss; S4. Perform weighted sampling based on multiple negative samples on the image-text pair dataset and calculate the weighted contrast loss of multiple negative samples; specifically, the steps include: If the anchor point is image modality data, a positive sample and several negative samples of text modality data are collected, the distance between the anchor point and the positive sample, and the distance between the anchor point and several negative samples are calculated, and the comparison weight values ​​of the positive sample and the negative sample are adjusted so that the distance between the anchor point and the positive sample in the common space is as close as possible and the distance between the anchor point and each negative sample in the common space is as far as possible; If the anchor point is text modal data, a positive sample and several negative samples of the image modal data are collected, the distance between the anchor point and the positive sample, and the distance between the anchor point and several negative samples are calculated, and the comparison weight values ​​of the positive sample and the negative sample are adjusted so that the distance between the anchor point and the positive sample in the common space is as close as possible and the distance between the anchor point and each negative sample in the common space is as far as possible; The following formula is used to calculate the weighted contrast loss L2 of multiple negative samples: Where w u is the weight when the anchor point is image modality data; f () is the similarity measurement function; u is the image modality anchor point; v+ is the positive sample text similar to the image modality anchor point u; N1 is the number of sampled negative samples that are not similar to the image modality anchor point u; v k is the kth negative sample sampled and dissimilar to the image modality anchor point u; w v is the weight when the anchor point is text modality data; v is the text modality anchor point; u+ is the positive sample image similar to the text modality anchor point v; N2 is the number of sampled negative samples that are not similar to the text modality anchor point v; u k is the kth negative sample sampled that is dissimilar to the text modality anchor v; S5. According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, the constructed cross-modal image-text retrieval initial model is optimized by an optimizer to obtain a cross-modal image-text retrieval model; S6. Use the cross-modal image-text retrieval model obtained in step S5 to perform actual image-text cross-modal retrieval.

2. A cross-modal image-text retrieval method according to claim 1, characterized in that The step S1 of obtaining a data set of image-text pairs and constructing an initial model for cross-modal image-text retrieval specifically includes the following steps: A. Obtaining a picture-text pair dataset: The picture-text pair dataset includes an image dataset and a text dataset; B. The constructed initial model of cross-modal image-text retrieval includes: extracting image feature vector sets from image datasets through convolutional neural networks; extracting text feature vector sets from text datasets through bag-of-words models; then projecting the image feature vector sets and text feature vector sets into a common space through a projection function; and finally performing similarity measurement and label classification in the common space.

3. A cross-modal image-text retrieval method according to claim 2, characterized in that Step S2 of processing the image-text pair data set obtained in step S1 to obtain a feature vector specifically includes the following steps: The image dataset in the image-text pair dataset obtained in step S1 is subjected to a convolutional neural network to extract image features, thereby obtaining an image feature vector set; the text dataset in the image-text pair dataset obtained in step S1 is subjected to a bag-of-words model to extract key semantic words, thereby obtaining a text feature vector set.

4. A cross-modal image-text retrieval method according to claim 3, characterized in that Step S3, in which the feature vector obtained in step S2 is projected through a function to obtain a feature vector of uniform dimension in a common space, performs label classification and calculates the label classification loss, specifically includes the following steps: a. Pass the image feature vectors in the image feature vector set obtained in step S2 through a three-layer fully connected layer network; b. Pass the text feature vectors in the text feature vector set obtained in step S2 through a three-layer fully connected layer network; c. Through the processing of steps a and b, the image feature vector and the text feature vector are located in the same real-valued space, and then parameters are shared through the same fully connected layer, and finally projected into the same low-dimensional public space for label classification; d. Use the following formula to calculate the label classification loss L1: Where n is the number of training samples; y i is the label for each training instance; p i (u i ) is the probability distribution of the generated image; p i (v i ) is the generated text probability distribution.

5. The method for cross-modal image-text retrieval according to claim 1, characterized in that The calculation of the distance between the anchor point and the positive sample, and between the anchor point and a number of negative samples, specifically calculates the cosine distance between the anchor point and the positive sample, and between the anchor point and a number of negative samples.

6. A cross-modal image-text retrieval method according to claim 1, characterized in that According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, step S5 optimizes the constructed cross-modal image-text retrieval initial model through an optimizer to obtain a cross-modal image-text retrieval model, which specifically includes the following steps: According to the label classification loss obtained in step S3 and the multi-negative sample weighted contrast loss obtained in step S4, the Adam optimizer is used to update and continuously optimize the parameters in the initial model of cross-modal image and text retrieval, so as to obtain the final cross-modal image and text retrieval model.

7. A cross-modal image-text retrieval method according to claim 5, characterized in that Step S6 uses the cross-modal image-text retrieval model obtained in step S5 to perform actual image-text cross-modal retrieval, which specifically includes the following steps: When the retrieval object is text modal data, the distance between the text modal data and each data in the image retrieval library is calculated, and according to the obtained distance, the best several image data are selected as the final retrieval result for feedback, thereby completing the actual image-text cross-modal retrieval; When the retrieval object is image modal data, the distance between the image modal data and each data in the text retrieval library is calculated, and according to the obtained distance, the optimal number of text data is selected as the final retrieval result for feedback, thereby completing the actual image-text cross-modal retrieval.

Images