A cross-media retrieval method and system fusing multi-granularity data and near neighbor data

Abstract

The application discloses a kind of fusion multi-granularity data and cross-media retrieval method and system of near neighbor data, comprising the following steps: respectively extracting the coarse-grained feature and fine-grained feature of image data and text data, respectively calculate the corresponding coarse-grained layer similarity and fine-grained layer similarity using ladder triple loss function;Respectively extract the near neighbor layer image feature of image data and the near neighbor layer text feature of text data, calculate the near neighbor layer similarity of image data and text data;The semantic category constraint of coarse-grained layer and fine-grained layer and the near neighbor relationship constraint of near neighbor layer are fused, and the cross-media similarity of fusion multi-granularity data and near neighbor data is obtained.The application uses improved ladder triple loss function and attention mechanism, respectively calculates the similarity of heterogeneous modal data on coarse-grained layer, fine-grained layer and near neighbor layer, obtains the cross-media similarity of fusion multi-granularity data and near neighbor data, improves the precision of cross-media similarity measurement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cross-media retrieval, and particularly relates to a cross-media retrieval method and system fusing multi-granularity data and neighbor data. BACKGROUND

[0002] With the advent of the big data era, multimedia data on the Internet is growing explosively, and media types are constantly enriched. Multimedia data of different modalities can be used for expression of similar content, resulting in increasing demand for cross-media retrieval. Cross-media retrieval refers to inputting query data of any media type by a user, and retrieving other media type data related to the query data. However, data of different modalities have different attributes, distribution and representation, and therefore, it is difficult to measure the similarity between data of different modalities, which is also the core problem of cross-media retrieval.

[0003] In real life, there is a "heterogeneous gap" between data of different modalities, that is, the representation of image, text and other different media data is inconsistent, resulting in difficulty in realizing unified representation and comprehensive utilization of multiple media data. However, data of different modalities naturally have cross-modal correlation in semantics, and research on human cognition also shows that the human brain recognizes the world by fusing vision, hearing and language. Therefore, how to integrate the human cognitive mode into cross-media retrieval is a key link to cross the "heterogeneous gap". At present, the common solution is to map data of different modalities to the same public subspace, and then select appropriate measurement methods according to the characteristics of the data to realize cross-media similarity measurement.

[0004] Most of the existing cross-media retrieval methods are for coarse-grained data, use convolutional neural networks to extract features of heterogeneous modal data, and realize cross-media similarity measurement based on semantic category constraints through traditional triplet loss functions, such as joint graph regularization method based on traditional machine learning method, and measurement method based on deep learning which optimizes all negative classes at the same time. However, the existing methods only consider coarse-grained data and do not fully consider and utilize fine-grained data and neighbor data, which affects the accuracy of cross-media similarity measurement, and the traditional triplet loss function brings the semantic related heterogeneous modal data as close as possible, and pushes away the semantic unrelated heterogeneous modal data with the same amplitude, that is, the relevance of the candidate samples and the query samples in the heterogeneous modal data is considered as bipolar, which also affects the accuracy of similarity measurement of data of different modalities. SUMMARY

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a cross-media retrieval method and system fusing multi-granularity data and neighbor data, which uses an improved ladder-type triplet loss function and an attention mechanism to respectively calculate the similarity of heterogeneous modal data on three data layers of a coarse-granularity layer, a fine-granularity layer and a neighbor layer, fully excavates the semantic complementarity of the heterogeneous modal data, and obtains the cross-media similarity of the fused multi-granularity data and neighbor data, thereby improving the precision of cross-media similarity measurement.

[0006] The technical scheme of the cross-media retrieval method fusing multi-granularity data and neighbor data provided by the present application is as follows:

[0007] The cross-media retrieval method fusing multi-granularity data and neighbor data comprises the following steps:

[0008] Coarse-granularity features of image data and text data are respectively extracted, and a coarse-granularity layer similarity of the image data and the text data is obtained by using a ladder-type triplet loss function;

[0009] Fine-granularity features of the image data and the text data are respectively extracted, and a fine-granularity layer similarity of the image data and the text data is obtained by using the ladder-type triplet loss function;

[0010] Neighbor layer image features of the image data and neighbor layer text features of the text data are respectively extracted, a neighbor image similarity and a neighbor text similarity are obtained by calculation, and the neighbor layer similarity of the image data and the text data is obtained by linear weighting of the two;

[0011] The semantic category constraint of the coarse-granularity layer and the fine-granularity layer and the neighbor relationship constraint of the neighbor layer are fused, and the cross-media similarity of the fused multi-granularity data and neighbor data is obtained.

[0012] Further, the coarse-granularity features of the image data and the text data refer to the features of the whole image as the coarse-granularity features of the image data and the features of the whole text as the coarse-granularity features of the text data.

[0013] Further, the ladder-type triplet loss function refers to the improvement of the traditional triplet loss function, according to the similarity of the negative example sample and the query sample, setting multiple thresholds to divide the similarity into multiple levels, and pushing the negative example sample away from the query sample at different amplitudes.

[0014] Further, the fine-grained features of the extracted image data and text data refer to that after the whole image is uniformly divided into several blocks, the features of each image block are extracted as the fine-grained features of the image data; all the words in the text data are input into the GRU, the attention mechanism is adopted to calculate the fine-grained attention weight of the image block to the word, the words with the attention weight greater than a set threshold are screened out, and the features of the screened words are taken as the fine-grained features of the text data.

[0015] Further, the calculation step of the near-neighbor image similarity includes that after the near-neighbor images of the test image and the text matched with the near-neighbor images are obtained, the attention weight of each near-neighbor image is obtained by using the attention mechanism, the text matched with the near-neighbor images is weighted processed by using the learned attention weight, and the sum of the similarities between the weighted text and the test text is taken as the near-neighbor image similarity of the test image.

[0016] Further, the calculation step of the near-neighbor text similarity includes that after the near-neighbor texts of the test text and the images matched with the near-neighbor texts are obtained, the attention weight of each near-neighbor text is obtained by using the attention mechanism, the images matched with the near-neighbor texts are weighted processed by using the learned attention weight, and the sum of the similarities between the weighted images and the test image is taken as the near-neighbor text similarity of the test text.

[0017] Further, the fusion of the semantic category constraints of the coarse-grained layer and the fine-grained layer and the near-neighbor relationship constraints of the near-neighbor layer to obtain the cross-media similarity of the fused multi-grained data and near-neighbor data refers to that the coarse-grained layer similarity, the fine-grained layer similarity and the near-neighbor layer similarity are linearly weighted and calculated to obtain the final cross-media similarity of the image data and the text data.

[0018] Another aspect of the present disclosure provides a technical scheme of a cross-media retrieval system fusing multi-grained data and near-neighbor data.

[0019] A cross-media retrieval system fusing multi-grained data and near-neighbor data includes:

[0020] The coarse-grained layer similarity calculation module extracts the coarse-grained features of the image data and the text data respectively, and calculates the coarse-grained layer similarity of the image data and the text data by using the stepwise triplet loss function.

[0021] The fine-grained layer similarity calculation module extracts the fine-grained features of the image data and the text data respectively, and calculates the fine-grained layer similarity of the image data and the text data by using the stepwise triplet loss function.

[0022] The neighbor layer similarity calculation module: the neighbor layer image features of the image data and the neighbor layer text features of the text data are extracted respectively, neighbor image similarity and neighbor text similarity are calculated and obtained, and the neighbor layer similarity of the image data and the text data is obtained after averaging;

[0023] The cross-media similarity calculation module: the semantic category constraints of the coarse-grained layer and the fine-grained layer and the neighbor relationship constraints of the neighbor layer are fused, and the cross-media similarity of the fused multi-grained data and the neighbor data is obtained.

[0024] The technical scheme of the computer-readable storage medium provided by the present disclosure is:

[0025] A computer-readable storage medium has a computer program stored thereon, and the program is executed by a processor to implement the steps in the cross-media retrieval method of fusing multi-grained data and neighbor data.

[0026] The technical scheme of the computer device provided by the present disclosure is:

[0027] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the steps in the cross-media retrieval method of fusing multi-grained data and neighbor data.

[0028] Compared with the prior art, the technical scheme of the present disclosure has the following beneficial effects:

[0029] The cross-media retrieval method of fusing multi-grained data and neighbor data provided by the present disclosure calculates the similarity of heterogeneous modal data on three data layers of coarse-grained layer, fine-grained layer and neighbor layer respectively, fully excavates the semantic complementarity of heterogeneous modal data, fuses the "semantic category constraints" of the coarse-grained layer and the fine-grained layer and the "neighbor relationship constraints" of the neighbor layer through linear weighting, and obtains the cross-media similarity of the fused multi-grained data and the neighbor data, thereby improving the precision of cross-media similarity measurement.

[0030] The bidirectional attention mechanism provided by the present disclosure is used to process fine-grained data and neighbor data, and the similarity of heterogeneous modal data on the fine-grained layer and the neighbor layer is calculated by using the mechanism, thereby improving the accuracy of cross-media similarity measurement in the two directions of "image retrieval text" and "text retrieval image".

[0031] The stepwise triple loss function provided by the present disclosure effectively improves the shortcomings of the traditional triple loss function, classifies the negative example samples according to the distance from the query sample, and sets different intervals, so as to more accurately depict the distance between heterogeneous modal data, and thereby improve the precision of cross-media similarity measurement. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification. The embodiments of the application, together with its

[0033] Figure 1 Structure framework diagram of the cross-media retrieval method of fusing multi-granularity data and neighbor data in the embodiment of the application;

[0034] Figure 2 Schematic diagram of the improved stepwise triplet loss function in the embodiment of the application;

[0035] Figure 3(a) is a PR curve diagram of image retrieval text in the Corel5K dataset in the embodiment of the application;

[0036] Figure 3(b) is a PR curve diagram of text retrieval image in the Corel5K dataset in the embodiment of the application;

[0037] Figure 4(a) is a PR curve diagram of image retrieval text in the Pascal Sentence dataset in the embodiment of the application;

[0038] Figure 4(b) is a PR curve diagram of text retrieval image in the Pascal Sentence dataset in the embodiment of the application. DETAILED DESCRIPTION

[0039] The present disclosure will be further described with reference to the drawings and embodiments.

[0040] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0041] It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. It should be noted that, as used in this specification and the appended claims, the singular form “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. It should also be noted that, as used in this specification and the appended claims, the term “or” is generally employed in its sense of “and / or” unless the context clearly dictates otherwise.

[0042] Embodiment One

[0043] The embodiment discloses a cross-media retrieval method and system fusing multi-granularity data and neighbor data, which respectively calculates the similarity of heterogeneous modal data on three data layers of a coarse-granularity layer, a fine-granularity layer and a neighbor layer by using an improved ladder triple loss function and an attention mechanism, fully excavates the semantic complementarity of the heterogeneous modal data, obtains the cross-media similarity of the multi-granularity data and the neighbor data, and improves the precision of cross-media similarity measurement.

[0044] Please refer to the accompanying Figure 1 The cross-media retrieval method fusing multi-granularity data and neighbor data comprises the following steps:

[0045] Step 1: coarse-granularity features of image data and text data are respectively extracted, and the coarse-granularity layer similarity of the image data and the text data is calculated by using a ladder triple loss function.

[0046] Specifically, first, the features of the whole image are extracted as the coarse-granularity features of the image data, and the features of the whole text are extracted as the coarse-granularity features of the text data.

[0047] In the embodiment, the heterogeneous modal data is image modal data and text modal data, and a cross-media data set composed of images and sentences is represented as Wherein, the data set D includes N images and N sentences, i p represents the image p, t q represents the sentence q.

[0048] If the sentence t q can accurately describe the visual content of the image i p , they constitute an "image-text pair". The samples in the same "image-text pair" belong to the same semantic category, and the samples not in the same "image-text pair" belong to different semantic categories.

[0049] The coarse-granularity layer extracts the features of the whole image and the whole sentence as the coarse-granularity features. Considering that the convolutional neural network can effectively extract the visual features of the image, under the Caffe framework, the AlexNet pre-trained on the ImageNet is used as the image feature encoder to extract the 4096-dimensional CNN features, which are mapped to 1024 dimensions to represent the coarse-granularity features of the image, denoted as In order to effectively utilize the text context information, the coarse-granularity layer quantizes all words by using One-Hot Encoding and inputs them into the GRU (Gated Recurrent Unit) to extract the hidden layer features as the coarse-granularity features of the text modal, denoted as

[0050] Then the coarse-grained layer similarity is calculated, which refers to the similarity between the whole image and the whole sentence. The visual feature vector of the query image , The text feature vector of the retrieval sentence , for the traditional triplet loss function, the image as a query sample, if the retrieved sentence and the image belong to the same semantic category, it is considered as a successful query; if the retrieved sentence and the image belong to different semantic categories, it is considered as a failed query. At the same time, N -p is defined as the set of all sentences belonging to different semantic categories with the query sample , thus the inequality can be obtained:

[0051] Sim(v p , h p )-Sim(v p , h q )>a (1)

[0052] The inequality can be converted into a triplet loss function:

[0053]

[0054] wherein, and are defined as follows:

[0055]

[0056]

[0057] wherein, represents the feature vector binary tuple of the image and the sentence under the same semantic category in the coarse-grained layer, represents the feature vector binary tuple of the image and the sentence under different semantic categories in the coarse-grained layer, and the parameter a represents the minimum interval between the two similarities.

[0058] Considering that the traditional triplet loss function only brings semantically related images and sentences as close as possible to each other, while semantically unrelated images and sentences are pushed away by the same amount, that is, for a query sample, only samples that completely match its "semantic category" are considered "related", while other samples, even if they have some similarity, are considered "unrelated", leading to a decrease in the accuracy of similarity measurement, in order to improve the accuracy of semantic information expression in cross-media retrieval, this invention improves on the basis of the traditional triplet loss function and proposes a stepped triplet loss function, which sets different intervals to treat "unrelated" samples differently.

[0059] Specifically, the schematic diagram of the stepped triplet loss function is as follows: Figure 2 As shown, the relevance of the four samples a, b, c, and d to the query image decreases sequentially. Here, a is a positive sample, and b, c, and d are negative samples. The traditional triplet loss pushes b, c, and d away from the query image by the same distance. However, the stepped triplet loss proposed in this application can push b, c, and d away from the query image by different magnitudes based on their similarity to the query image, thereby effectively improving the accuracy of similarity measurement.

[0060] In this embodiment, the traditional triplet loss function (i.e., Equation 2) is used to calculate the image. and sentences The similarity is defined by setting the threshold set as {θ1, θ2, ..., θ...} L-1 The similarity is then divided into L levels based on L-1 thresholds. Then, the dataset is further processed using the defined L-1 thresholds. Divide into L subsets The data set representing the distance from the query sample pushed away by the interval α1 is used to extend the constraints of the triplet loss into a chain of inequalities:

[0061]

[0062] in, α1,...,α L This represents the similarity margin of disjoint subsets. For example, belonging to... The samples are pushed away from the query samples by a distance of a1+a2+a3, while those belonging to The samples will only be pushed away from the query sample by a distance of a1. For simplicity, taking a ladder loss consisting of three subsets (i.e., L=3) as an example, the formal definition of the ladder triple is given:

[0063]

[0064] in, These are the first step loss of the coarse-grained layer, the second step loss of the coarse-grained layer, and the third step loss of the coarse-grained layer, respectively.

[0065]

[0066] wherein β1, β2 and β3 represent the weights of and satisfy β1+β2+β3=1; and represent the union from to and the union from to ; when β2=β3=0, formula 4 will degenerate into the traditional triplet loss function.

[0067] Step 2: Extract the fine-grained features of the image data and the text data respectively, and calculate the fine-grained layer similarity of the image data and the text data by using the ladder triplet loss function.

[0068] Considering that the attention mechanism focuses on relevant data and suppresses irrelevant data, this feature is consistent with the requirement of cross-media retrieval for fine-grained data, so the attention mechanism is applied to cross-media retrieval.

[0069] In this embodiment, the fine-grained layer uses the attention mechanism to mine the semantic information contained in the image blocks and the words, and filters out important image blocks and words from the multiple image blocks of an image and the words describing the semantic information of the image blocks.

[0070] After dividing an image into k×k blocks, input the image blocks into the AlexNet network and train to obtain the visual features of the image blocks, then map the visual features to 1024-dimensional vectors through the fully connected layer, and the feature vector set of n f image blocks is represented as wherein n f =k×k represents the number of image blocks in an image; for the fine-grained data of the text mode, the processing manner is similar to that of the coarse-grained data of the text mode in the coarse-grained layer, input all m words in a sentence into the GRU to obtain a feature vector set composed of feature vectors of all words, filter out the words with higher attention weights, and the feature vector set composed of feature vectors of the filtered words is represented as wherein m f represents the number of filtered words in a sentence.

[0071] Specifically, the feature vector of the pth image block of the image is represented as the feature vector of the qth word in the sentence matched with the image is represented as The calculation method of the fine-grained attention weight of the image block p to the word q is as follows:

[0072] ​

[0073] where, denotes the similarity between image patch p and word q.

[0074] The fine-grained layer similarity refers to the similarity between image patch in image and word h f in sentence t f , which is calculated as shown in the following formula 7:

[0075]

[0076] where, denotes the attention weight of the fine-grained layer, m f denotes the number of words in sentence t f that are screened out for calculation, that is, only words with attention weight greater than 1 / m are allowed to participate in similarity calculation.

[0077] After the important word screening process using the attention mechanism, S nearest neighbor image patches are selected from the n f image patches of the image, and the objective function is defined as follows:

[0078]

[0079] where, denote the feature vector pair of image and sentence under the same semantic category in the fine-grained layer, denotes the similarity between image patch and word under the same semantic category; denote the feature vector pair of image patch and word under different semantic categories in the fine-grained layer, denotes the similarity between image patch and word under different semantic categories; parameter a represents the minimum interval between the two similarities.

[0080] Similar to the coarse-grained layer, the stepwise constraint condition of the fine-grained layer data is:

[0081]

[0082] where,

[0083] Based on the above analysis, the stepwise loss function of the fine-grained layer is defined as follows:

[0084]

[0085] where, are the first step loss of the fine-grained layer, the second step loss of the fine-grained layer and the third step loss of the fine-grained layer, respectively;

[0086]

[0087] wherein β1, β2 and β3 represent the weights of and β1+β2+β3=1, α1,...,α L is the similarity interval of disjoint subsets in the fine-grained layer.

[0088] Step 3: Extract the near-neighbor layer image features of the image data and the near-neighbor layer text features of the text data respectively, calculate the near-neighbor image similarity and the near-neighbor text similarity, and combine to obtain the near-neighbor layer similarity of the image data and the text data.

[0089] Considering the similarity between the image i p and the sentence t q paired with it should be higher than the similarity between the image i p and other sentences, therefore, the present application takes this basic fact as the basic principle for measuring the similarity of the sentence t q and other sentences, and calculates the similarity of the image and the sentence by using the "image near-neighbor relationship" and the "sentence near-neighbor relationship".

[0090] The coarse-grained layer and the fine-grained layer take semantic categories as the constraint condition, while the near-neighbor layer takes the near-neighbor relationship as the constraint condition, specifically, the solution of the similarity between the image i p and the sentence t q is divided into two steps:

[0091] First, the visual modality near-neighbor of the image i p is obtained, and the sentences matched with each visual near-neighbor image are obtained, the attention mechanism is used to weight the matched sentence features, and the similarity between the weighted sentence features and the sentence t q is calculated, which is called "image near-neighbor similarity".

[0092] Second, the text modality near-neighbor of the sentence t q is obtained, the images matched with each near-neighbor are obtained, and the weights of the images are obtained by using the attention mechanism, and then the similarity between the weighted image features and the image i p is calculated, which is called "sentence near-neighbor similarity".

[0093] In this embodiment, let i train and t train represent the image training set and the sentence training set respectively, i test and t test represent the image test set and the sentence test set respectively. The feature extraction method of the near-neighbor layer is the same as that used in the coarse-grained layer and the fine-grained layer, input the AlexNet network and train to obtain the image features of the near-neighbor layer, n rThe set of feature vectors of the nearest neighbor layer images is represented as follows: Meanwhile, the set of text feature vectors in the nearest neighbor layer is represented as Where m r Indicates the number of neighboring sentences.

[0094] Image attention similarity calculation consists of two steps: the first step is to calculate the nearest neighbor images in the training set. text t train Attention weights of images from neighboring layers The product of these is used as the final text feature vector.

[0095] The attention weight of the nearest neighbor layer image The calculation method is as follows:

[0096] For test images Obtained using the nearest neighbor strategy mr Training set of nearest neighbor images Based on this, calculate the attention weights of the nearest neighbor layer images.

[0097] in, Represents images in the training set Images in the test set Similarity between them Represents training set images eigenvectors, Represents test set images eigenvectors.

[0098] The second step is to calculate the similarity between the test set images and the final text feature vectors, using the images in the test set as the basis for the similarity calculation. Image attention similarity:

[0099]

[0100] in, t represents the attention weights between the nearest neighbor images in the training set and the images in the test set. train and t test These represent the test set text and the training set neighbor text, respectively.

[0101] Similarly, the text attention similarity can be calculated, which consists of two steps: the first step is to calculate the nearest neighbor texts t in the training set. train Image Attention weights for sentences in the nearest neighbor layer The product of these is used as the final image feature vector.

[0102] The nearest neighbor layer sentence attention weight The calculation method is as follows:

[0103] For the test text m nearest neighbor texts are obtained using the nearest neighbor strategy r (q = 1, 2,..., m r ), and on this basis, the nearest neighbor layer text attention weight

[0104]

[0105] wherein, denotes the similarity between the sentence in the training set and the sentence in the test set, denotes the feature vector of the training set sentence , and denotes the feature vector of the test set sentence .

[0106] The second step is to calculate the similarity between the test set text and the final image feature vector as the text attention similarity of the test set text:

[0107]

[0108] wherein, denotes the training set nearest neighbor image and the test set text attention weight, v train , and v test denote the test set image and the training set nearest neighbor image, respectively.

[0109] Finally, the nearest neighbor sentence similarity and the nearest neighbor image similarity are weighted to obtain the similarity between the nearest neighbor layer image and the sentence, and the specific formula is:

[0110]

[0111] Step 4: The semantic category constraints of the coarse-grained layer and the fine-grained layer and the nearest neighbor relationship constraints of the nearest neighbor layer are fused to obtain the cross-media similarity of the fused multi-grained data and the nearest neighbor data.

[0112] The coarse-grained data, fine-grained data and nearest neighbor data of the visual modal and text modal are used to fully exploit the semantic complementarity of the two modal data, and the cross-media similarity of the fused multi-grained data and the nearest neighbor data is obtained. Specifically, the cross-media similarity calculation method of image i and sentence t is as follows:

[0113] Sim(i, t) = a · Sim coarse (i c ,t c ) + β · Sim fine (i​​f t f )+γ·Sim neighbor (i r ,t r ) (17)

[0114] wherein, Sim coarse denotes the coarse-grained layer similarity, Sim fine denotes the fine-grained layer similarity, and Sim neighbor denotes the near-neighbor layer similarity; i c and t c denote the image and sentence of the coarse-grained layer; i f and t f denote the image and sentence of the fine-grained layer; i r and t r denote the image and sentence of the near-neighbor layer; and α, β, γ denote the weight of the three types of similarity.

[0115] In order to verify the effectiveness of the cross-media retrieval method for fusing multi-granularity data and near-neighbor data proposed in the present application, in the present embodiment, the present application takes the Corel5K dataset and the Pascal Sentence dataset as examples, uses cosine distance to measure the similarity, verifies the effectiveness of the algorithm by using MAP and PR curve, and compares with the following nine methods to verify the improvement of the effect of the cross-media retrieval method.

[0116] wherein, the Corel5K dataset contains 5000 pictures, each picture in the dataset is assigned with 1-5 labeled words, the training set includes 374 labeled words, the test set includes 260 labeled words, of which 500 pictures are used for testing and verification, and the remaining images are used for training.

[0117] The Pascal Sentence dataset contains 1000 pictures, each picture is accompanied by 5 sentences describing the visual content. The dataset includes 800 training pictures, 100 verification pictures and 100 test pictures. And the sentences in the Corel5K dataset are sentences composed of a group of words, while the sentences in the Pascal Sentence dataset are sentences in natural language concepts.

[0118] The nine comparative methods are as follows:

[0119] Random: search for images or text by generating random numbers to verify whether the method is effective;

[0120] CCA: find the linear projection of the two largest related random vectors;

[0121] CMCP: propose a method for simultaneously using positive and negative correlation cross-media retrieval;

[0122] HSNN: Propose a similarity measure based on the nearest neighbor idea for heterogeneous modalities;

[0123] JGRHML: Use graph regularization to learn semantic information through label propagation based on heterogeneous metric;

[0124] JRL: Use semi-supervised regularization and sparse regularization methods, and use semantic information to learn public space;

[0125] JSSFL: Learn the projection matrix of each modality, map the multi-modal data to the public space, and preserve the relationship between modalities and modalities;

[0126] CCL: Use hierarchical network to fuse coarse-grained and fine-grained data to learn cross-modal correlation;

[0127] S2UPG: Use a semi-supervised cross-media feature learning framework to model multi-modal multi-granularity data.

[0128] MAP (mean average precision) is an evaluation index reflecting the performance of the system in retrieving all relevant texts of the picture (or retrieving all relevant pictures of the text), which can solve the single-point value limitation of accuracy and recall. The size is related to the ranking of the retrieval effect, and the more relevant documents retrieved by the system (rank is higher), the higher the MAP should be.

[0129] Specifically, according to the optimal experimental effect, the triple interval parameters a1, a2, a3 of the Corel5K dataset coarse-grained layer are set to 0.2, 0.1, 0.05; The triple interval parameters a1, a2, a3 of the fine-grained layer are set to 0.2, 0.08, 0.02, and the parameters α, β, γ are set to 1, 6, 12.

[0130] For the Pascal Sentence dataset, the triple interval parameters a1, a2, a3 of the coarse-grained layer are set to 0.18, 0.12, 0.06, and the triple interval parameters a1, a2, a3 of the fine-grained layer are set to 0.16, 0.08, 0.02, and the parameters α, β, γ of the similarity measure framework are set to 1, 4, 25.

[0131] Table 1 MAP evaluation results of Corel5K dataset

[0132]

[0133] Table 2 MAP evaluation results of Pascal Sentence dataset

[0134]

[0135]

[0136] From the experimental results of Table 1, it can be seen that the method of the present application obtains the best performance in the Corel5K dataset compared with the methods, and the average MAP of two retrieval directions reaches 0.4944, effectively verifying the effectiveness of the fusion of the semantic relationship and the near neighbor relationship. Among the remaining methods, CCA, CMCP and HSNN have relatively poor effects; S2UPG and CCL have better effects, and the average MAP of two retrieval directions is 0.4269 and 0.4384 respectively, because these two methods use global and local semantic information, which effectively improves the retrieval performance.

[0137] From the experimental results of the Pascal Sentence dataset given in Table 2, it can be seen that the method of the present application is still better than other comparison methods on this dataset. Among other methods, the performance of CCL is better, and the average MAP reaches 0.5686; in addition, it can also be seen that the effects of HSNN, JSSFL and other non-multiple granularity data fusion methods are not ideal, which shows that multiple granularity data fusion has important significance for effectively mining the complementary relationship between different granularity data and then improving the cross-media retrieval performance.

[0138] From the PR curves of "image retrieval text" and "text retrieval image" given in Figures 3(a), 3(b), 4(a) and 4(b), it can be seen from the experimental results shown in Figures 3 and 4 that the PR curve coverage area of the method of the present application is higher than that of other methods, and the retrieval performance is better than that of other methods.

[0139] Through the comparison of the above experimental results, it is further proved that whether it is image retrieval related text or text retrieval related image, the cross-media retrieval method proposed in the present application performs better than other cross-media retrieval methods, which fully verifies the effectiveness of the method.

[0140] In addition, in order to verify the effect of each part of the present application, the effectiveness of the scheme of the present application is further compared and verified through the ablation experiment.

[0141] Table 3 MAP ablation experiment of Corel5K and Pascal Sentence dataset

[0142]

[0143]

[0144] Among them, "no step loss" represents the experimental results of removing the step loss function part, and the coarse-grained layer, the fine-grained layer and the near neighbor layer respectively represent the experimental results of using only coarse-grained data, fine-grained data and near neighbor data.

[0145] From Table 3, it can be seen that the accuracy of the similarity measurement of fusing three data layers is higher than that of any data layer. This shows that making full use of the context information and the proximity relationship under the attention mechanism can more effectively improve the precision of cross-media retrieval; and without using the stepwise loss, the MAP value is generally about 2 percentage points lower on the Corel5K or Pascal Sentence dataset, which shows that the stepwise loss function can accurately distinguish the differences between negative example samples, thereby effectively improving the accuracy of cross-media retrieval.

[0146] In this embodiment, the experimental results show that for the typical cross-media retrieval task of "image-text mutual retrieval", the method described in the application performs better than other methods on multiple datasets; and the ablation experiment further proves that the method can make full use of the complementary relationship between multi-granularity data and proximity data, and the similarity measurement method of fusing three data layers is better than each single data layer and the fusion of any two data layers.

[0147] Embodiment Two

[0148] The purpose of this embodiment is to provide a cross-media retrieval system fusing multi-granularity data and proximity data, comprising:

[0149] A coarse-grained layer similarity calculation module: coarse-grained features of image data and text data are extracted respectively, and a stepwise triplet loss function is used to calculate the coarse-grained layer similarity of the image data and the text data;

[0150] A fine-grained layer similarity calculation module: fine-grained features of image data and text data are extracted respectively, the fine-grained features of the text data are determined by using an attention mechanism, and a stepwise triplet loss function is used to calculate the fine-grained layer similarity of the image data and the text data;

[0151] A proximity layer similarity calculation module: proximity layer image features of image data and proximity layer text features of text data are extracted respectively, proximity image similarity and proximity text similarity are calculated, and the proximity layer similarity of the image data and the text data is obtained;

[0152] A cross-media similarity calculation module: the semantic category constraints of the coarse-grained layer and the fine-grained layer and the proximity relationship constraints of the proximity layer are fused, and the cross-media similarity fusing multi-granularity data and proximity data is obtained.

[0153] Embodiment Three

[0154] The purpose of this embodiment is to provide a computer-readable storage medium.

[0155] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the cross-media retrieval method of fusing multi-granularity data and proximity data as described above.

[0156] Embodiment four

[0157] The purpose of this embodiment is to provide a computer device.

[0158] A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the cross-media retrieval method of fusing multi-granularity data and proximity data as described above when executing the program.

[0159] The steps and methods involved in the above embodiments two to four correspond to embodiment one, and the detailed description can refer to the relevant description part of embodiment one. The term "computer readable storage medium" should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying a set of instructions for execution by a processor and causing the processor to perform any of the methods in the present application.

[0160] Those skilled in the art should understand that each module or step of the present application described above can be implemented by a general computer device, alternatively, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0161] The above description is only the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0162] The above describes the specific embodiments of the present application in combination with the accompanying drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A cross-media search method of fusing multi-granularity data and proximity data, characterized by, The method comprises the following steps: Respectively extract coarse-grained features of image data and text data, and calculate coarse-grained layer similarity of the image data and the text data by using a ladder type triplet loss function; Respectively extract fine-grained features of the image data and the text data, and calculate fine-grained layer similarity of the image data and the text data by using the ladder type triplet loss function; The fine-grained features of the image data and the text data are extracted by dividing the whole image into several blocks, extracting the features of each image block as the fine-grained features of the image data, inputting all the words in the text data into a GRU, and calculating the fine-grained attention weight of the image block to the word by using an attention mechanism, and screening the words with the attention weight greater than a set threshold, and extracting the features of the screened words as the fine-grained features of the text data; The ladder type triplet loss function is improved from a traditional triplet loss function, and multiple thresholds are set according to the similarity between negative example samples and query samples to divide the similarity into multiple levels, and the negative example samples are pushed away from the query samples at different amplitudes; wherein , , are the first, second and third coarse-grained layer step losses, respectively; wherein, , and respectively represent , , the weight of the i-th feature, and satisfy ; and respectively represent the union from to and the union from to ; , , , represent the number of similarity intervals of disjoint subsets; represents the data set pushed away from the query sample by the interval ; the sample of is pushed away from the query sample by the distance of ; Respectively extract neighbor layer image features of the image data and neighbor layer text features of the text data, calculate neighbor image similarity and neighbor text similarity, and linearly weight the two to obtain neighbor layer similarity of the image data and the text data; The calculation steps of the neighbor image similarity comprise the following steps: after obtaining the neighbor images of a test image and the text matched with the neighbor images, the attention weight of each neighbor image is obtained by using an attention mechanism, the text matched with the neighbor images is weighted by using the learned attention weight, and the sum of the similarity between the weighted text and the test text is calculated as the neighbor image similarity of the test image; The semantic category constraint of the coarse-grained layer and the fine-grained layer and the near neighbor relationship constraint of the near neighbor layer are fused to obtain cross-media similarity of fused multi-grained data and near neighbor data; image and sentence The cross-media similarity calculation method is as follows: wherein, denotes coarse-grained layer similarity, denotes fine-grained layer similarity, denotes near-neighbor layer similarity; and denotes coarse-grained layer image and sentence; and denotes fine-grained layer image and sentence; and denotes near-neighbor layer image and sentence; , , denotes the weight of the three types of similarity.

2. The cross-media search method of fusing multi-granularity data and proximity data according to claim 1, characterized in that, The coarse-grained features of the image data and the text data are extracted by extracting the features of the whole image as the coarse-grained features of the image data and extracting the features of the whole text as the coarse-grained features of the text data.

3. The cross-media search method of claim 1, wherein the step of fusing multi-granularity data and proximity data comprises the steps of: calculating a weight of each of the multi-granularity data and the proximity data; and fusing the multi-granularity data and the proximity data by using the calculated weight. The calculation steps of the neighbor text similarity comprise the following steps: after obtaining the neighbor texts of a test text and the images matched with the neighbor texts, the attention weight of each neighbor text is obtained by using an attention mechanism, the images matched with the neighbor texts are weighted by using the learned attention weight, and the sum of the similarity between the weighted images and the test image is calculated as the neighbor text similarity of the test text.

4. The cross-media search method of fusing multi-granularity data and proximity data according to claim 1, wherein, The fusion of the semantic category constraints of the coarse-grained layer and the fine-grained layer and the neighbor relationship constraints of the neighbor layer to obtain the cross-media similarity of the fused multi-grained data and the neighbor data is linearly weighted calculation of the coarse-grained layer similarity, the fine-grained layer similarity and the neighbor layer similarity to obtain the final cross-media similarity of the image data and the text data.

5. A cross-media search system that fuses multi-granularity data and proximity data, characterized by, The method comprises the following steps: A coarse-grained layer similarity calculation module is configured to respectively extract coarse-grained features of image data and text data, and calculate coarse-grained layer similarity of the image data and the text data by using a ladder type triplet loss function; The fine-grained layer similarity calculation module extracts fine-grained features of the image data and the text data respectively, and calculates fine-grained layer similarities of the image data and the text data by using a ladder-type triplet loss function. The fine-grained features of the image data and the text data are extracted by dividing the whole image into several blocks and extracting features of each image block as the fine-grained features of the image data, and by inputting all words in the text data into a GRU, using an attention mechanism to calculate fine-grained attention weights of the image blocks on the words, and screening words with attention weights greater than a set threshold, and extracting features of the screened words as the fine-grained features of the text data. The ladder-type triplet loss function is an improved traditional triplet loss function, which sets multiple thresholds to divide the similarity into multiple levels according to the similarity between the negative example sample and the query sample, and pushes the negative example sample away from the query sample at different amplitudes. wherein , , are the first, second and third coarse-grained layer step losses, respectively; wherein, , and respectively represent , , the weight of the similarity interval, and satisfy ; and respectively represent the union from to and the union from to ; , , , represent the similarity interval of disjoint subsets; represents the data set pushed away from the query sample by the interval ; the sample of is pushed away from the query sample by the distance of ; The near neighbor layer similarity calculation module extracts near neighbor image features of the image data and near neighbor text features of the text data respectively, calculates near neighbor image similarities and near neighbor text similarities, and obtains the near neighbor layer similarity of the image data and the text data by averaging. The calculation steps of the near neighbor image similarity include obtaining near neighbor images of a test image and text matching the near neighbor images, using an attention mechanism to obtain attention weights of each near neighbor image, using the learned attention weights to weight the text matching the near neighbor images, and calculating the sum of similarities between the weighted text and the test text as the near neighbor image similarity of the test image. The cross-media similarity calculation module fuses the semantic category constraints of the coarse-grained layer and the fine-grained layer and the neighbor relationship constraints of the neighbor layer to obtain cross-media similarity of the fused multi-grained data and neighbor data. Specifically, the cross-media similarity calculation method of the image and the sentence is as follows: wherein, represents coarse-grained layer similarity, represents fine-grained layer similarity, represents near-neighbor layer similarity; and represents images and sentences of the coarse-grained layer; and represents images and sentences of the fine-grained layer; and represents images and sentences of the near-neighbor layer; , , represents weights of the three types of similarity.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the cross-media retrieval method of fusing multi-granularity data and near neighbor data according to any one of claims 1-4.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the cross-media retrieval method of fusing multi-granularity data and near neighbor data according to any one of claims 1-4.

Images