Cross-modal fine-grained retrieval method based on modal specific features and modal sharing features

Abstract

The invention provides a cross-modal fine-grained retrieval method based on modal specific features and modal sharing features. The target of the cross-modal fine-grained retrieval task is that a user gives any sample as a query sample, and the system retrieves to obtain each modal sample related to the query sample. On one hand, a branch network is used for extracting modal specific features of each modal, so that feature information of each modal is fully utilized; and on the other hand, the same network is used for extracting common features of the modes so as to enhance the relation between the modes, and the model learns the generality and the relation between data of different modes. In this way, specific information of each mode and shared information of different modes are utilized at the same time, semantic information contained in the high-dimensional semantic space vector is increased, heterogeneous gaps and semantic gaps between samples are reduced, the clustering capacity of the model for the high-dimensional space vector is enhanced, and the cross-modal retrieval capacity of the model is further improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to technologies such as deep learning, cross-modal retrieval, and fine-grained retrieval, and in particular to a cross-modal fine-grained retrieval method based on modal-specific features and modal shared features. Background Art

[0002] With the development of social science and technology, the four channels of video, image, text and audio have become the main forms for humans to understand the world and communicate with each other. The rapid growth of multimodal data has brought about a large number of application requirements for cross-modal retrieval, whose goal is to achieve mutual retrieval of cross-modal content. Compared with single-modal retrieval, cross-modal retrieval has higher technical challenges due to the large changes between modalities. However, current cross-modal retrieval tasks are usually focused on coarse granularity, which is far from meeting the needs of practical applications. In contrast, fine-grained retrieval has greater application needs and research value, both in industry and academia. Therefore, cross-modal fine-grained retrieval has become an important research direction, and more cross-modal fine-grained retrieval theories and technologies need to be developed.

[0003] Existing cross-modal retrieval methods mainly focus on the modality of image-text pairs, while there are few studies on the four modalities of image, video, audio and text. Multimodal retrieval means that the user gives a sample of any modality as a query sample, and the system retrieves and feedbacks samples of each modality that belong to the same category as the query sample. Multimodal retrieval generally requires that the number of modalities is greater than or equal to two, which also brings great difficulty to the experiment. Feature extraction is the basis of cross-modal fine-grained retrieval. Convolutional neural networks are used to extract image features, and recurrent neural networks or Transformers are used to extract text features. These models can obtain high-dimensional and rich feature representations. Multimodal fusion is one of the key technologies. The fusion of features of different modalities can be achieved through simple splicing, weighted fusion or more complex fusion models (such as attention mechanisms). Modality alignment is another important aspect, which studies how to align features of different modalities in the semantic space. Common methods include using contrast loss to minimize the distance between samples of the same type and maximize the distance between samples of different types. Cross-modal embedding maps data from different modalities into the same embedding space so that similar content is as close as possible in the space. This can be achieved through a neural network with shared weights or a generative adversarial network. The retrieval strategy can be content-based, which is based on the content of the image or text. It usually requires a similarity measure for the features (such as cosine similarity or Euclidean distance). It can also be semantic-based, which captures the semantic relationship between different modalities by building a higher-level semantic model to improve the accuracy of the retrieval. In terms of evaluation indicators, precision and recall are used to measure the quality of retrieval results, especially in fine-grained retrieval to ensure high relevance. Mean average precision (mAP) comprehensively considers the accuracy and completeness of the retrieval results and is suitable for multi-category retrieval tasks. In traditional multimodal retrieval models, different neural networks are generally used to extract feature vectors for different modalities, or a backbone network is used to extract feature vectors of different modalities at the same time. The former focuses on using modality-specific information, but it is difficult to extract the connections between modalities and the commonalities of samples of different modalities. The latter focuses on extracting the connections and commonalities between modalities, but the commonalities and connections are only a small part of all data, resulting in a large amount of effective modality-specific information loss. This method uses a backbone network to extract the modality-shared information of each modality, and uses a branch network to extract modality-specific information. The modality-shared information can focus on the commonalities and connections between modalities, and the modality-specific information can focus on the specific information of each modality itself. The two types of information are combined to obtain a feature vector containing rich semantic information, and through various loss functions, it is combined into a method suitable for cross-modal fine-grained retrieval tasks. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of heterogeneous gap and semantic gap between modalities in existing methods, and to provide a cross-modal fine-grained retrieval network based on modality-specific features and modality-shared features.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] 1) Convert the audio modality data into a spectrogram using short-time Fourier transform, crop the image modality data according to the existing bounding box labels, extract key frames from the video modality data by extracting one frame every ten frames, convert the text modality into a one-dimensional word vector through word embedding, and then splice it into an image-like matrix form through convolution operations.

[0007] 2) Use MixNet-S network to extract features from the image modality, video modality, and audio modality data that have been converted into pictures, as well as the processed text modality data, and finally obtain the modality-specific features of the four modalities, Mix(f I ), Mix(f V ), Mix(f A ), Mix(f T ).

[0008] 3) Use ResNet50 to extract the shared features of the four modal data, and finally obtain the modal shared features Res(f ALL ).

[0009] 4) Concatenate the modality-specific features and modality-shared features to obtain the final fusion feature F MIX .

[0010] 5) Fusion feature F MIX Input a linear classifier for classification, and use the ohemloss loss function and the discriminant loss function to optimize the model effect.

[0011] 6) Enter any category of any modality in the classified results, and retrieve other modal samples of the same category based on the classification to achieve cross-modal fine-grained retrieval.

[0012] Specifically, the step 1) includes:

[0013] For image data, since we only need the part of the whole image related to the retrieval object, in order to eliminate the interference of background noise, we crop the image. Cropping is performed according to the pixel coordinates of the retrieval object to obtain the part containing only the object.

[0014] For video data, we extract ten frames of images from each video as input samples of the video modality.

[0015] For audio data, we use short-time Fourier transform to convert it into the corresponding spectrogram as the input sample of the audio modality.

[0016] For text data, we convert it into a one-dimensional word vector through word embedding, and then concatenate it into an image-like matrix form through convolution.

[0017] Further, step 2) comprises:

[0018] For the matrix information of the four modes that have been processed, the MixNet-S network is used for feature extraction. The MixNet-S network combines convolution kernels of different sizes, which can extract features in different receptive fields, thereby effectively capturing local and global information.

[0019] Further, step 3) comprises:

[0020] We choose ResNet50 network to extract the common features of the four modalities. That is, the matrix results of the information preprocessing of the four modalities are uniformly used for feature extraction using ResNet50 network. ResNet50 is a deep residual network that solves the gradient vanishing and degradation problems in deep neural network training by introducing "residual blocks", so that the model can be trained deeper and more abundant features can be extracted, making the network have better classification effect.

[0021] Further, step 5) comprises:

[0022] Since the problem addressed by the present invention is the fine-grained cross-modal retrieval of 200 different bird species belonging to the same category of birds, the classification task is very heavy. In order to enable the model to distinguish the categories to which fine-grained samples belong, this method first performs supervised learning through the ohemloss function, and by setting the crossentropyloss function value of easy-to-classify samples to 0, the model is more focused on learning the detailed differences between similar samples.

[0023] That is, based on the original cross-entropy function, only a part of the samples with larger loss is learned for each batch of samples, so that the model is more focused on extracting the differences between different samples:

[0024]

[0025] Among them, y is the sample label value, p i is the network output value, l(x) is the indicator function, if the loss value is greater than the threshold, the value is 1, otherwise the value is 0.

[0026] At the same time, in order to prevent the backbone network and the branch network from learning too similar features, this method uses a discriminant loss function to maximize the prediction probability of the correct category and minimize the prediction probability of the wrong category to improve the model's ability to distinguish different categories:

[0027]

[0028] in is the feature similarity function, which is used to calculate the similarity between the modal shared eigenvector and the modal specific eigenvector, u i refers to the mode-shared eigenvector, v i refers to the modal-specific eigenvector; δ(x) is the sigmoid function, that is, ζ represents the threshold value, which is used to control the sensitivity of the loss. Therefore, the total loss function will be defined as L total =αL ohem +βL dis .

[0029] After the above steps 1) to 6), for a use case of a certain category of a certain modality input by the user, the system will retrieve and feedback use cases of other modalities belonging to the same category as the use case.

[0030] The beneficial effects of the present invention are as follows:

[0031] 1) The input of the image modality is cropped according to the bounding box to eliminate the interference of background noise.

[0032] 2) Convert image, video and audio modal data information into pictures, making network processing more convenient and the structure more unified.

[0033] 3) The data information of the text modality is converted into an image-like matrix form, so that the text modality information can be processed together with the information of the other three modalities, the shared features of the modalities can be extracted, and the connection and commonality between the modalities can be enhanced.

[0034] 4) Extract modal shared information and modal specific information at the same time, fuse them together and then use them together as the input of the classification task. In this way, the model considers both local and global information, which is conducive to improving the classification accuracy of the model.

[0035] 5) Use the ohemloss function network to learn the detailed information between fine-grained samples to improve the classification accuracy of fine-grained samples. Use the discriminateloss function to prevent the backbone network and branch network from learning too similar features, so that the network can better utilize the specific features of each modality and the common features of the modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0038] Figure 2 It is a schematic diagram of the model framework of the present invention.

[0039] Figure 3 It is the statistical data of the data set used in the experiment of the present invention.

[0040] Figure 4 This is a performance comparison chart of the present invention and other methods on the PKUFGXmedia dataset. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention proposes a cross-modal fine-grained retrieval network based on modality-specific features and modality-shared features. The main process of the method is as follows: Figure 1 The specific implementation process is as follows:

[0043] 1) Convert the audio modality data into a spectrogram using short-time Fourier transform, crop the image modality data according to the existing bounding box labels, extract key frames from the video modality data by extracting one frame every ten frames, convert the text modality into a one-dimensional word vector through word embedding, and then splice it into an image-like matrix form through convolution operations.

[0044] 2) Use MixNet-S network to extract features from the image modality, video modality, and audio modality data that have been converted into pictures, as well as the processed text modality data, and finally obtain the modality-specific features of the four modalities, Mix(f I ), Mix(f V ), Mix(f A ), Mix(f T ).

[0045] 3) Use ResNet50 to extract the shared features of the four modal data, and finally obtain the modal shared features Res(f ALL ).

[0046] 4) Concatenate the modality-specific features and modality-shared features to obtain the final fusion feature F MIX .

[0047] 5) Fusion feature F MIX Input a linear classifier for classification, and use the ohemloss loss function and the discriminant loss function to optimize the model effect.

[0048] 6) Enter any category of any modality in the classified results, and retrieve other modal samples of the same category based on the classification to achieve cross-modal fine-grained retrieval.

[0049] Specifically, the step 2) includes:

[0050] This method uses the MixNet-S network to extract features from branch networks. MixNet-S is a more efficient convolutional neural network architecture that aims to ensure model performance while reducing the consumption of computing resources. Its design is inspired by the combination of multiple convolution kernels, including multi-scale convolution, depth-separable convolution, channel attention mechanism, feature fusion, and residual connection. Multi-scale convolution refers to the use of multiple convolution kernel sizes (such as 1*1, 3*3, 5*5, etc.) in MixNet-S to process input feature maps in parallel. In this way, the network can capture information of different spatial resolutions, thereby enhancing the network's feature extraction capabilities. Depth-separable convolution refers to the fact that MixNet-S divides the standard convolution operation into two steps: depth-wise convolution and point-by-point convolution.

[0051] Among them, the deep convolution is to perform convolution operations independently for each channel. In the deep convolution, each input channel has a corresponding convolution kernel, and the size of the convolution kernel is usually 3x3 or 5x5. It has the advantages of independent convolution, few parameters, and high computational efficiency. For example, the input feature map is Where H refers to the height of the feature map, W refers to the width of the feature map, and C in Refers to the number of input channels, so the feature map in K refers to the depth of the convolution kernel, and k is the size of the convolution kernel.

[0052] Point-by-point convolution refers to the use of 1*1 convolution kernel to convolve the feature map. Its function is to combine and integrate the features of multiple feature channels output by deep convolution. It has the characteristics of fusion channel information, few parameters, and high flexibility. where Y∈R H×W×Cout, K refers to the point-by-point convolution kernel, C out Refers to the number of output channels.

[0053] Further, step 3) comprises:

[0054] This method uses the ResNet50 network to extract features from the backbone network. The ResNet50 network is a variant of the deep residual network with a depth of 50 layers. It is mainly used for image classification tasks. The core of the ResNet50 network lies in its residual block. Each residual block contains two 3*3 convolutional layers. After each convolutional layer, there is a batch normalized ReLU activation function. Then, the input is directly added to the output of the convolutional layer through a shortcut connection to form residual learning. Shortcut connections are divided into identity shortcuts and convolution shortcuts. The identity shortcut refers to directly adding the input and output when the dimensions of the input and output are the same; the convolution shortcut refers to using 1x1 convolution for linear transformation when the dimensions of the input and output do not match, so that their dimensions are consistent, and then the addition operation is performed. The ResNet50 network structure is as follows: the input layer receives a 224x224 RGB image, then uses a 7*7 convolution kernel with a stride of 2, followed by a 3*3 maximum pooling layer. Next comes the residual module, which contains multiple residual blocks and is divided into four stages. The number of channels and residual blocks in each stage is different. Stage 1 has 64 channels and 3 residual blocks, stage 2 has 128 channels and 4 residual blocks, stage 3 has 256 channels and 6 residual blocks, and stage 4 has 512 channels and 3 residual blocks. In the last stage, global average pooling is used to reduce the dimension of the feature map, and finally a fully connected layer is used to output the classification result. The use of the ResNet50 network can solve the gradient vanishing problem, making information more efficient in the network. At the same time, due to the 50-layer network, the expression ability of the model is enhanced.

[0055] Further, step 4) includes:

[0056] After obtaining the multimodal feature vector, it is necessary to effectively utilize the modal-specific features and modal-shared features to fuse and map the features to d 3 Common space, and clustering is performed through loss function. Assuming modal specific feature extraction method m∈{I,V,A,T}, to obtain modality-specific features in represents the i-th sample pair under the m mode, P m represents the trainable parameters of the m-mode-specific branch network, d 1 The vector dimension representing the modality-specific features, I represents the image modality, V represents the video modality, A represents the audio modality, and T represents the text modality. Assuming the modality-shared feature extraction method Get the modal shared features Where P represents the trainable parameters of the feature sharing backbone network, d 2 The vector dimension representing the modal shared feature. Then the modal specific feature vector is directly concatenated with the modal shared feature vector to obtain the modal fusion feature. Finally, map it to the public space where d 3 Represents the dimension of the common space feature vector.

[0057] After the above steps 1) to 6), for a use case of a certain category of a certain modality input by the user, the system will retrieve and feedback use cases of other modalities belonging to the same category as the use case.

[0058] Experimental verification

[0059] 1. Dataset

[0060] The data set used in the present invention is the only public data set FG-Xmedia for fine-grained retrieval across four modalities. The data set contains data in four modalities: image, video, audio, and text. The image data comes from CUB-200-2011, which is the most widely used fine-grained image classification data set. It contains 11,788 images of 200 subcategories, belonging to the same basic coarse-grained category as "Bird", wherein the training set contains 5,994 images, and the test set contains 5,794 images. Each image annotation contains an image-level subcategory label, the bounding box of the object, 15 partial positions, and 312 binary attributes. The video source is YouTube Birds, which is a new fine-grained video data set. The same category division method as CUB-200-2011 is used, wherein the training set includes 12,666 videos, and the test set includes 5,684 videos. The audio and text data are obtained from professional websites according to the divided categories. The above four modalities together constitute the public data set FG-Xmedia.

[0061] 2. Implementation details

[0062] The present invention mainly uses ResNet50 as the backbone network and MixNet-S as the branch network. After preprocessing, the data is extracted by these two networks to extract feature vectors, which are then mapped to the public space through a 512-dimensional fully connected layer. The public space vector is then passed through a 200-dimensional fully connected layer, and supervised learning is performed with categories as labels. The entire program code is written in Pytorch, and the graphics card is RTX-3090. During training, the initial learning rate is set to 3e-3, and the learning rate decay strategy uses the ReduceLROnPlateau strategy provided by the PyTorch framework, and training is performed for 200 rounds.

[0063] 3. Comparative experiment

[0064] The present invention compares the proposed method with some representative models, and the present invention names the proposed method as MS2.

[0065] (1) ACMR: A novel Adversarial Cross-Modal Retrieval (ACMR) method that seeks an effective common subspace based on adversarial learning.

[0066] (2) CMDN: A cross-media multi-deep network that exploits complex cross-media associations through hierarchical learning, learns rich cross-media correlations in two stages, and finally obtains shared representations through a stacked network working approach.

[0067] (3) JRL: A novel feature learning algorithm for cross-media data, namely joint representation learning (JRL), can jointly explore relevant information and semantic information under a unified optimization framework.

[0068] (4) GSPH: A simple yet effective general hashing framework that can be applied to all different situations while preserving the semantic distance between data points. This method first simultaneously learns the optimal hash codes of the two modalities to preserve the semantic similarity between data points, and then learns a hash function to map features to hash codes.

[0069] (5) MHTN: A modality-backward hybrid transfer network (MHTN) aims to achieve knowledge transfer from a unimodal source domain to a cross-modal target domain and learn cross-modal joint representations.

[0070] (6) FGCrossNet: A unified deep model that learns four types of media simultaneously without processing them separately. Three constraints are considered together: the classification constraint ensures the learning of discriminative features of fine-grained subcategories, the center constraint ensures the compactness of features of the same subcategory, and the sorting constraint ensures the sparsity of features of different subcategories.

[0071] (7) DBFC-Net: A novel dual-branch fine-grained cross-media network (DBFC-Net) that exploits media-specific information to build common features through a unified framework. It also designs an effective distance metric for fine-grained cross-media retrieval.

[0072] (8) SAFGCM: An attention space training method to learn the common representation of different media data. Specifically, a local self-attention layer is used to learn the common attention space between different media data.

[0073] For fair comparison, this paper uses a unified evaluation index for multimodal fine-grained retrieval tasks, namely mean average precision (mAP). In this experiment, mean average precision refers to first calculating the average accuracy of query samples in each category, and then calculating the average of the average accuracy of 200 categories to obtain mAP.

[0074] At the same time, in order to more comprehensively demonstrate the effect of cross-modal fine-grained retrieval, we set up a total of 18 evaluation indicators, including the individual retrieval scores of each modality for the other three modalities, represented as I→V, I→A, I→T, V→I, V→A, V→T, A→I, A→V, A→T, T→I, T→V, T→A, and the average of the above scores. It also includes the common retrieval scores of each modality for all other modalities, namely I→ALL, V→ALL, A→ALL, T→ALL, and the average of the above scores. Observation Figure 4 From the data, we can see that our algorithm has significantly improved performance compared with other methods, and has improved by more than 100% compared to the FGCrossNet algorithm in the I→T, T→V, A→T, V→T, and T→ALL scenarios. Overall, the average retrieval mAP value of one-to-one modal retrieval has increased by 24% compared to the FGCrossNet algorithm, and the average retrieval mAP value of one-to-multimodal retrieval has increased by 36% compared to the FGCrossNet algorithm. This proves the effectiveness of MS2 in cross-modal fine-grained retrieval, which can fully integrate information from different modalities, enhance information interaction of fine-grained objects in different modalities, reduce differences between different modalities, make full use of the rich semantic information of the text modality, reduce information loss, etc., so as to better improve the retrieval effect.

Claims

1. A cross-modal fine-grained retrieval method based on modality-specific features and modality-shared features, characterized by: The following steps are involved: 1) The audio modality data is converted into a spectrogram using short-time Fourier transform, the image modality data is cropped according to the existing bounding box labels, the video modality data is extracted by extracting key frames in the manner of extracting one frame every ten frames, and the text modality is converted into a one-dimensional word vector through word embedding, and then concatenated into an image-like matrix form through convolution operation; 2) The image modality, video modality and audio modality data that have been converted into pictures, as well as the processed text modality data are extracted using the MixNet-S network, and finally the modality-specific features of the four modalities are obtained, Mix(f I ), Mix(f V ), Mix(f A ), Mix(f T ); 3) Use ResNet50 to extract the shared features of the four modal data, and finally obtain the modal shared features Res(f ALL );4) Combine the modality-specific features with the modality-shared features to obtain the final fusion feature F MIX ; 5) Fusion feature F MIX Input the linear classifier for classification, and use the ohem loss function and the discriminant loss function to optimize the model effect; 6) Input any category of any modality in the classified results, retrieve other modal samples of the same category according to the classification, and realize cross-modal fine-grained retrieval.

2. The cross-modal fine-grained retrieval method based on modality-specific features and modality-shared features according to claim 1 is characterized in that: Step 1) includes: using the existing bounding box labels of the image modality to crop the input data to eliminate the interference of background noise. At the same time, in order to use MixNet-S to process video and audio information, the video is frame-extracted, and one frame is extracted every ten frames as a representative of the ten frames. The audio is processed using short-time Fourier transform, and they are all converted into images as the input of MixNet-S. The text modality is converted into a one-dimensional word vector through word embedding, and then spliced ​​into an image-like matrix form through convolution operations.

3. The cross-modal fine-grained retrieval method based on modality-specific features and modality-shared features according to claim 1 is characterized in that: Step 2) includes: for the processed matrix information of the four modes, use the MixNet-S network to extract features. The MixNet-S network combines convolution kernels of different sizes, so that features can be extracted on different receptive fields, thereby effectively capturing local and global information.

4. The cross-modal fine-grained retrieval method based on modality-specific features and modality-shared features according to claim 1 is characterized in that: Step 3) includes: extracting modal shared features of the four modalities using a ResNet50 network. ResNet50 is a deep convolutional neural network that includes a residual learning framework. Each residual block of ResNet50 contains a jump connection, which directly adds the input to the output, helping to alleviate the gradient disappearance and promote the effective transmission of messages. At the same time, residual learning allows the network to learn the identity mapping, thereby simplifying the network learning process.

5. The cross-modal fine-grained retrieval method based on modality-specific features and modality-shared features according to claim 1 is characterized in that: Step 5) includes: according to the fusion feature F obtained in step 4) MIX , and use it as the input of a linear classifier to classify each category of data, so that the retriever can directly search in the public space of the category of the corresponding label according to the category label of the input modal information and output the information of other modalities belonging to the same category.

6. The cross-modal fine-grained retrieval method based on modality-specific features and modality-shared features according to claim 1 is characterized in that: The corresponding step 6) includes: a cross-modal fine-grained retriever performs retrieval based on the classification results obtained in step 5). The classification in step 5) does not consider the modality but only classifies by category. Therefore, use cases of the same category in different modalities will be grouped together. After the user inputs a category of a certain modality, the system will retrieve and feedback use cases of other modalities that belong to the same category as the input sample, thereby realizing the cross-modal fine-grained retrieval task.

Images