Cross-modal retrieval method and storage medium based on subspace adaptive spacing

Abstract

The present invention discloses a cross-modal retrieval method and storage medium based on subspace adaptive spacing, comprising the following steps: data preprocessing, data partitioning into a training set and a test set, and extracting the original high-dimensional features of the data; inputting the original high-dimensional features into a network model to obtain the common features of the image and text and the corresponding predicted label information; using the common features and label information to calculate the adaptive spacing loss of samples of different categories in each modality, then combining the attention mechanism to focus on the category information in the image and text to enhance the discriminability of different categories, and finally calculating the invariance loss between different modalities; then optimizing the loss function through backpropagation to iterate the network model until convergence; using the converged network model to calculate the common features of all images and texts; and finally calculating the similarity between the query data features and the common features, sorting and returning the results. The accuracy of cross-modal retrieval using the present invention is higher than that of cross-modal retrieval using existing methods.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-modal retrieval method and storage medium using supervised subspace adaptive spacing. Background Art

[0002] In recent years, with the rapid development of the internet and the ubiquity of smart devices and social networks, multimedia data has exploded on the internet. Recent data shows that 300 hours of video are uploaded to YouTube every minute; hundreds of millions of images are uploaded daily to photo-sharing sites like Flickr and Snapchat; and on social media sites like Facebook and Twitter, users share massive amounts of multimedia data, including text, images, and videos. These different types of data often describe the same object or event. For example, web pages often combine text, images, or videos to express the same event or theme. As the volume of data on the internet continues to grow, data types are becoming increasingly diverse, and users are increasingly interested in more diverse data modalities. User search needs are also shifting from single-modality to cross-modality. Modality refers to the form in which data is expressed, including text, images, video, and audio. Cross-modal search involves searching data in at least two modalities, typically using one modality as a query to retrieve relevant data in another. By identifying potential connections between data in different modalities, relatively accurate cross-matching can be achieved.

[0003] Measuring the content relevance between data from different modalities is a core challenge in cross-modal retrieval. Traditionally, the triplet loss function used in cross-modal retrieval is based on a fixed margin value, with an adjustable margin hyperparameter α. However, the paper "Online low-rank similarity function learning with adaptive relative margin for cross-modal retrieval" notes that the margin values ​​between samples of different categories vary, making a fixed margin value unreasonable. Therefore, the paper proposes an adaptive margin value. However, this paper only considers the impact of single-modal samples on the adaptive margin, which undoubtedly reduces the correlation between multiple modalities and leads to reduced retrieval accuracy. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the above-mentioned existing technologies and propose a cross-modal retrieval method based on supervised subspace adaptive spacing to solve the technical problem of low retrieval accuracy in existing cross-modal hash retrieval methods.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A cross-modal retrieval method based on subspace adaptive spacing includes the following steps:

[0007] Data preprocessing, dividing the data into training set and test set, and extracting the original high-dimensional features of the data;

[0008] Input the original high-dimensional features into the network model to obtain the common features of the image and text output by the model and the corresponding predicted label information;

[0009] Use common features and label information to calculate the adaptive distance loss of different categories of samples in each modality. Combined with the attention mechanism, it focuses on the category information in images and text to enhance the discriminability of different categories and calculates the invariance loss between samples of different modalities.

[0010] Then optimize the loss function through back propagation to iterate the network model until the model converges;

[0011] Use the converged network model to calculate the common features of all images and texts;

[0012] Finally, the similarity between the query data features and the common features is calculated and the results are sorted and returned.

[0013] Furthermore, the data preprocessing, dividing the data into training set and test set, and extracting the original high-dimensional features of the data, includes the following steps:

[0014] S1a, collect data image, text data And label information in represents the original sample of the i-th image, represents the original sample of the i-th text, y i represents the unique hot encoding of the i-th sample, c represents the number of categories, and then the training set and test set are divided;

[0015] S1b. For the image training set data, use the VGG19 pre-trained model on image-net for fine-tuning, and then use the fine-tuned model to extract all the original features of the image;

[0016] S1c. For the text training set data, the Word2Vec model is used as a universal dictionary to build a text matrix for each text, which is then input into the sentence CNN network for pre-training. The pre-trained model is then used to extract all the original features of the text.

[0017] Furthermore, the input original high-dimensional features are fed into the network model to obtain the common features of the image and text output by the model and the corresponding predicted label information, including the following steps:

[0018] S2, the original features of the image and the original features of the text in step S1 are learned through the subspace network to obtain the features of the same dimension Q = (q1, q2, ...q n ), where q i Represents each subspace image feature, S=(s1,s2,…s n ) where s i Represents the text features of each subspace;

[0019] S3: For the subspace features in step S2, use a linear classifier to predict the categories of images and texts. in represents the one-hot encoding of the image prediction label, in Represents the one-hot encoding of text prediction labels.

[0020] Furthermore, the common features and label information are used to calculate the adaptive distance loss of different categories of samples in each modality, and the attention mechanism is combined to focus on the category information in the image and text to enhance the discriminability of different categories, and to calculate the invariance loss between samples of different modalities, specifically including:

[0021] S4, for the subspace text and image features obtained in step S2, assume there is a text feature v i and image feature k i , then calculate v i Texts belonging to the same category Similarity Similarly calculate the source and in is the cosine function;

[0022] S5. For the subspace image feature q in step S2 i , calculate the text features of the same category subspace Similarity Text features of different categories of subspaces Similarity The same calculation Combined with the adaptive spacing function to construct text-to-image and image-to-text triple loss functions

[0023] S6. For each image-text pair After step S2, the corresponding subspace image and text features (qi ,s i ), calculate the similarity of the same pair of image text and the similarity between unpaired image texts Using triplet loss constraint and Similarity distance

[0024] S7. For the label predicted in step S3, use the F norm to constrain the image to predict the label P m With the true label Y and the text prediction label P t The distance from the true label Y Use triplet loss to enhance the accuracy of image and text category prediction

[0025] S8,

[0026] The attention mechanism consists of a text self-attention mechanism and an image self-attention mechanism. For the text self-attention mechanism, the original input S is mapped to the query space Q, key space K and value space V respectively through a 2048-dimensional linear layer with three ReLU activation functions. Then, the scaled dot product attention mechanism formula f(Q, K, V) is used to calculate the image attention feature M and the text attention feature N. The text-to-image and image-to-text ternary loss functions are constructed in combination with the adaptive spacing function.

[0027]

[0028] The text-to-image and image-to-text triplet loss functions as follows:

[0029]

[0030]

[0031] Among them, s * Indicates s i ,s j ,q * Indicates q i ,q j ;∈ represents hyperparameter;

[0032] The loss function as follows:

[0033]

[0034] Among them, α is a hyperparameter;

[0035] The loss and as follows:

[0036]

[0037]

[0038] Among them, β is a hyperparameter, Represents the predicted value corresponding to the position of the true label of the i-th image sample, Represents the predicted value corresponding to the position of the i-th image sample except the true label; Represents the predicted value corresponding to the position of the true label of the i-th text sample, Represents the predicted value corresponding to the position of the i-th text sample except the true label;

[0039] The attention mechanism text to image and image to text triple loss function as follows:

[0040]

[0041] Furthermore, the process of optimizing the loss function by backpropagation to iterate the network model until the model converges includes the following steps:

[0042] S8, Joint and Get the total target loss function And the target loss function Perform optimization and model training to obtain the subspace depth network parameter w r ;

[0043] The objective loss function as follows:

[0044]

[0045] Among them, λ1, λ2, λ3 and λ4 are hyperparameters.

[0046] Furthermore, the method of using the converged network model to calculate the common features of all images and texts includes the following steps:

[0047] S9, calculating subspace features of image query data and text query data;

[0048] S9a inputs the text query data into the above network model to obtain the subspace features of the text query data;

[0049] S9b inputs the image query data into the above network model to obtain the image query data subspace features.

[0050] Furthermore, the similarity between the query data features and the common features is calculated and the results are sorted and returned, specifically including:

[0051] For text queries, the cosine similarity between the subspace features of the text query data and the subspace features of the image training data is calculated; then the similarities are sorted and the sorted image data is returned;

[0052] For image queries, the cosine similarity between the subspace features of the image query data and the subspace features of the text training data is calculated, and then the similarities are sorted and the sorted text data is returned.

[0053] Furthermore, the adaptive spacing loss function is as follows:

[0054]

[0055] Among them, ξ1 and ξ2 are hyperparameters, Calculate the F-norm between the similarity of positive samples and negative samples in different modal samples, Calculate the F-norm between the similarity of positive samples and negative samples in the image modality, Calculate the F-norm between the similarity of positive samples and negative samples in the text modality.

[0056] Furthermore, the subspace network in step S2 consists of two fully connected layers containing 1024 hidden units and is shared by the image subnetwork and the text subnetwork. The dimensions learned through the subspace are all 1024 dimensions.

[0057] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the above method.

[0058] As can be seen from the above technical solution, the cross-modal retrieval method based on subspace adaptive spacing of the present invention achieves higher accuracy than existing cross-modal retrieval methods. Compared with existing technologies, the present invention can effectively utilize triple deep networks to improve the accuracy of cross-modal retrieval.

[0059] Specifically, the advantages of the present invention are as follows:

[0060] 1. This paper proposes an adaptive margin-supervised cross-modal structure to learn a universal representation of data from different modalities, thereby reducing cross-modal heterogeneity;

[0061] 2. The relative similarity of image-text pairs of the present invention is used to learn modality invariance;

[0062] 3. This paper proposes a new adaptive margin method to learn appropriate margin constraints;

[0063] 4. Experimental results on widely used benchmark datasets show that the performance of our method is better than that of existing cross-modal retrieval methods, which demonstrates the effectiveness of our approach. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic flow chart of the method of the present invention;

[0065] Figure 2 It is a schematic diagram of the model structure of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0067] like Figure 1 As shown, the cross-modal retrieval method based on subspace adaptive spacing described in this embodiment includes the following steps:

[0068] Data preprocessing, dividing the data into training set and test set, and extracting the original high-dimensional features of the data;

[0069] Input the original high-dimensional features into the network model to obtain the common features of the image and text output by the model and the corresponding predicted label information;

[0070] Use common features and label information to calculate the adaptive distance loss of different categories of samples in each modality. Combined with the attention mechanism, it focuses on the category information in images and text to enhance the discriminability of different categories and calculates the invariance loss between samples of different modalities.

[0071] Then optimize the loss function through back propagation to iterate the network model until the model converges;

[0072] Use the converged network model to calculate the common features of all images and texts;

[0073] Finally, the similarity between the query data features and the common features is calculated and the results are sorted and returned.

[0074] The following are detailed descriptions:

[0075] The data preprocessing includes dividing the data into training set and test set, and extracting the original high-dimensional features of the data, including the following steps:

[0076] 1) Data preprocessing:

[0077] 1a) Collect data images and text data And label information in represents the original sample of the i-th image, represents the original sample of the i-th text, y i Represents the unique hot encoding of the i-th sample, c represents the number of categories, and then the training set and test set are divided.

[0078] 1b) For the image training set data, use the VGG19 model pre-trained on image-net for fine-tuning, and then use the fine-tuned model to extract all image original features.

[0079] 1c) For the text training set data, the present invention uses the Word2Vec model as a universal dictionary to construct a text matrix for each text, which is then input into the sentence CNN network for pre-training, and then the pre-trained model is used to extract all the original features of the text.

[0080] The input original high-dimensional features enter the network model to obtain the common features of the image and text output by the model and the corresponding predicted label information, wherein the network model is as follows Figure 2 As shown, the following steps are included:

[0081] 2) The original features of the image and the original features of the text in step 1) are learned through the subspace network to obtain the features of the same dimension Q = (q1, q2, ...q n ), where q i Represents each subspace image feature, S=(s1,s2,…s n ) where s i Represents the text features of each subspace.

[0082] 3) For the subspace features in step 2), the present invention uses a linear classifier to predict the category of the image and text in represents the one-hot encoding of the image prediction label, in Represents the one-hot encoding of text prediction labels.

[0083] The method uses common features and label information to calculate the adaptive distance loss of different categories of samples in each modality, combines the attention mechanism to focus on the category information in pictures and texts to enhance the discriminability of different categories, and calculates the invariance loss between samples of different modalities, specifically including

[0084] 4) For the subspace text and image features obtained in step 2), assume that the present invention has a text feature v i and image feature k i Then the present invention can calculate v iTexts belonging to the same category Similarity Similarly, we can calculate the provenance and in Considering the influence of different modal samples on the adaptive distance, the present invention uses the F-norm to measure the similarity distance of different modal samples as the adaptive distance function.

[0085] 5) For the subspace image feature q in step 2) i , the present invention can calculate the subspace text features of the same category Similarity Text features of different categories of subspaces Similarity Similarly, we can calculate Combined with the adaptive spacing function in step 4) to construct the text-to-image and image-to-text triple loss functions

[0086] 6) For each image-text pair After step 2), the present invention can obtain the corresponding subspace image and text features (q i ,s i ), the present invention can calculate the similarity of the same pair of image text and the similarity between unpaired image texts In order to ensure that the distribution of the same pair of text images in the subspace is similar, the present invention adopts the triple loss constraint and Similarity distance

[0087] 7) For the predicted label in step 3), in order to make the predicted label close to the real label, the present invention uses the F norm to constrain the image prediction label P m With the true label Y and the text prediction label P t The distance from the true label Y Use triplet loss to enhance the accuracy of image and text category prediction

[0088] The process of optimizing the loss function by backpropagation to iterate the network model until the model converges includes the following steps:

[0089] 8) The attention mechanism consists of a text self-attention mechanism and an image self-attention mechanism, wherein the original input S of the text self-attention mechanism is mapped to the query space Q, key space K and value space V respectively through a 2048-dimensional linear layer with three ReLU activation functions, and then the image attention feature M and the text attention feature N are calculated using the scaled dot product attention mechanism formula f(Q, K, V), and the text-to-image and image-to-text ternary loss functions are constructed in combination with the adaptive spacing function.

[0090]

[0091] The method of using the converged network model to calculate the common features of all images and texts includes the following steps:

[0092] 9) Calculate the subspace features of image query data and text query data.

[0093] 9a) Inputting the text query data into the above network model to obtain the subspace features of the text query data.

[0094] 9b) Inputting the image query data into the above network model to obtain the image query data subspace features.

[0095] Finally, the similarity between the query data features and the common features is calculated and the results are sorted and returned, including:

[0096] 10) Get the query data retrieval results:

[0097] For text queries, the cosine similarity between the subspace features of the text query data and the subspace features of the image training data is calculated. The similarities are then sorted and the sorted image data is returned.

[0098] For image queries, the cosine similarity between the subspace features of the image query data and the subspace features of the text training data is calculated. The similarities are then sorted and the sorted text data is returned.

[0099] In this embodiment, the original feature dimension of the image extracted in step 1b) is 4096 dimensions.

[0100] The original feature dimension of the text extraction in step 1c) is 300 dimensions.

[0101] Step 2) The subspace network consists of two fully connected layers with 1024 hidden units and is shared by the image subnetwork and the text subnetwork. The dimensions learned by the subspace network are all 1024 dimensions.

[0102] Step 4) The adaptive spacing function is as follows:

[0103]

[0104] Among them, ξ1 and ξ2 are hyperparameters, Calculate the F-norm between the similarity of positive samples and negative samples in different modal samples, Calculate the F-norm between the similarity of positive samples and negative samples in the image modality, Calculate the F-norm between the similarity of positive samples and negative samples in the text modality.

[0105] Step 5) The text-to-image and image-to-text ternary loss functions as follows:

[0106]

[0107] Among them, s * Indicates s i ,s j ,q * Indicates q i ,q j ∈ represents a hyperparameter.

[0108] Step 6) The loss function as follows:

[0109]

[0110] Here, α is a hyperparameter.

[0111] Step 3) The linear classifier is related to the number of categories in the data set.

[0112] Step 7) The loss and as follows:

[0113]

[0114]

[0115] Among them, β is a hyperparameter, Represents the predicted value corresponding to the position of the true label of the i-th image sample, Represents the predicted value corresponding to the position of the i-th image sample except the true label. Represents the predicted value corresponding to the position of the true label of the i-th text sample, Represents the predicted value corresponding to the position of the i-th text sample except the true label;

[0116]

[0117] Step 8) Combine and Get the total target loss function And the target loss function Perform optimization and model training to obtain the subspace depth network parameter w r ;

[0118] The objective loss function as follows:

[0119]

[0120] Among them, λ1, λ2, λ3 and λ4 are hyperparameters.

[0121] The optimization method uses the Adam optimizer, and the model training is performed in batches, with each batch containing 100 samples.

[0122] In another aspect, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of any of the above methods.

[0123] On the other hand, the present invention further discloses a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any of the above methods.

[0124] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any one of the methods in the above embodiments.

[0125] It is understandable that the system provided by the embodiment of the present invention corresponds to the method provided by the embodiment of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts of the above method.

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0127] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A cross-modal retrieval method based on subspace adaptive spacing, characterized in that: The following steps are included: Step 1: Data preprocessing: divide the data into training set and test set, and extract the original high-dimensional features of the data; Step 2: Input the original high-dimensional features into the network model to obtain the common features of the image and text output by the model and the corresponding predicted label information, thereby obtaining the subspace text and image features; Step 3: Use common features and label information to calculate the adaptive gap loss for samples of different categories in each modality. Combined with the attention mechanism, focus on the category information in the image and text to enhance the discriminability of different categories, and calculate the invariance loss between samples of different modalities. Step 4: Optimize the loss function through back propagation to iterate the network model until the model converges; Step 5: Use the converged network model to calculate the common features of all images and texts; Step 6: Finally, calculate the similarity between the query data features and the common features and sort and return the results; Step three is as follows: S4, for the subspace text and image features obtained in step S2, assume there is a text feature v i and image feature k i , then calculate v i Texts belonging to the same category Similarity Similarly calculate the source and in is the cosine function; S5. For the subspace image feature q in step S2 i , calculate the text features of the same category subspace Similarity Text features of different categories of subspaces Similarity The same calculation Combined with the adaptive spacing function to construct text-to-image and image-to-text triple loss functions S6. For each image-text pair After step S2, the corresponding subspace image and text features (q i ,s i ), calculate the similarity of the same pair of image text and the similarity between unpaired image texts Using triplet loss constraint and Similarity distance S7. For the label predicted in step S3, use the F norm to constrain the image to predict the label P m With the true label Y and the text prediction label P t The distance from the true label Y Use triplet loss to enhance the accuracy of image and text category prediction S8. The attention mechanism consists of a text self-attention mechanism and an image self-attention mechanism. The original input S of the text self-attention mechanism is mapped to the query space Q, key space K and value space V respectively through a 2048-dimensional linear layer with three ReLU activation functions. Then, the scaled dot product attention mechanism formula f(Q, K, V) is used to calculate the image attention feature M and the text attention feature N. The text-to-image and image-to-text ternary loss functions are constructed in combination with the adaptive spacing function. Among them, the joint and Get the total target loss function And the loss function Perform optimization and model training to obtain the subspace depth network parameter w r , loss function as follows: Among them, λ1, λ2, λ3 and λ4 are hyperparameters, is the text-to-image and image-to-text triplet loss function, is the similarity distance loss, The distance between the image prediction label and the true label and the text prediction label and the true label, is the triplet loss used to enhance the accuracy of image and text category prediction. It is the ternary loss function for text-to-image and image-to-text based on the attention mechanism; in, as follows:

2. The cross-modal retrieval method based on subspace adaptive spacing according to claim 1, characterized in that: The data preprocessing is to divide the data into training set and test set, and extract the original high-dimensional features of the data. The following steps are included: S1a, collect data image, text data And label information in represents the original sample of the i-th image, Represents the original sample of the i-th text, y i represents the unique hot encoding of the i-th sample, c represents the number of categories, and then the training set and test set are divided; S1b. For the image training set data, use the VGG19 pre-trained model on image-net for fine-tuning, and then use the fine-tuned model to extract all the original features of the image; S1c. For the text training set data, the Word2Vec model is used as a universal dictionary to build a text matrix for each text, which is then input into the sentence CNN network for pre-training. The pre-trained model is then used to extract all the original features of the text.

3. The cross-modal retrieval method based on subspace adaptive spacing according to claim 2, characterized in that: The input original high-dimensional features are fed into the network model to obtain the common features of the image and text output by the model and the corresponding predicted label information, including the following steps: S2, the original features of the image and the original features of the text in step S1 are learned through the subspace network to obtain the features of the same dimension Q = (q1, q2, ...q n ), where q i Represents each subspace image feature, S=(s1,s2,…s n ) where s i Represents the text features of each subspace; S3: For the subspace features in step S2, use a linear classifier to predict the categories of images and texts. in represents the one-hot encoding of the image prediction label, in Represents the one-hot encoding of text prediction labels.

4. The cross-modal retrieval method based on subspace adaptive spacing according to claim 3, characterized in that: The text-to-image and image-to-text triplet loss functions as follows: Among them, s * Indicates s i ,s j ,q * Indicates q i ,q j ;∈ represents hyperparameter; The loss function as follows: Among them, α is a hyperparameter; The loss and as follows: Among them, β is a hyperparameter, Represents the predicted value corresponding to the position of the true label of the i-th image sample, Represents the predicted value corresponding to the position of the i-th image sample except the true label; Represents the predicted value corresponding to the position of the true label of the i-th text sample, Represents the predicted value corresponding to the position of the i-th text sample except the true label.

5. The cross-modal retrieval method based on subspace adaptive spacing according to claim 1, characterized in that: The converged network model is used to calculate the common features of all images and texts. The following steps are included: S9, calculating subspace features of image query data and text query data; S9a inputs the text query data into the above network model to obtain the subspace features of the text query data; S9b inputs the image query data into the above network model to obtain the image query data subspace features.

6. The cross-modal retrieval method based on subspace adaptive spacing according to claim 1, characterized in that: Finally, the similarity between the query data features and the common features is calculated and the results are sorted and returned, including: For text queries, the cosine similarity between the subspace features of the text query data and the subspace features of the image training data is calculated; then the similarities are sorted and the sorted image data is returned; For image queries, the cosine similarity between the subspace features of the image query data and the subspace features of the text training data is calculated, and then the similarities are sorted and the sorted text data is returned.

7. The cross-modal retrieval method based on subspace adaptive spacing according to claim 1, characterized in that: The adaptive spacing loss function is as follows: Among them, ξ1 and ξ2 are hyperparameters, Calculate the F-norm between the similarity of positive samples and negative samples in different modal samples, Calculate the F-norm between the similarity of positive samples and negative samples in the image modality, Calculate the F-norm between the similarity of positive samples and negative samples in the text modality.

8. The cross-modal retrieval method based on subspace adaptive spacing according to claim 1, characterized in that: The subspace network in step S2 consists of two fully connected layers containing 1024 hidden units and is shared by the image subnetwork and the text subnetwork. The dimensions learned through the subspace are all 1024 dimensions.

9. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is caused to perform the steps of the method according to any one of claims 1 to 8.

Images