Image-text cross-modal retrieval method, device and storage medium based on spherical hashing

Abstract

The present invention relates to the field of information retrieval technology, and in particular to a cross-modal image and text retrieval method, device and storage medium based on spherical hashing. In the present invention, the spherical von Mises-Fisher distribution is introduced into cross-modal hashing, uncertainty information is explored to guide the hashing process, and a discriminative modality-invariant hash code is generated. At the same time, in order to retain the semantic information of the sample-specific mode, a random spherical von Mises-Fisher loss function is proposed to achieve the alignment of different modes and semantic embeddings in the form of probability distribution.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval technology, and in particular to a method, device and storage medium for cross-modal image and text retrieval based on spherical hashing. Background Art

[0002] Generally, content-based hash retrieval can be divided into unimodal hashing and cross-modal hashing. Cross-modal hashing can be divided into supervised and unsupervised methods according to whether it relies on annotation information. In general, unsupervised cross-modal hashing usually relies on a pre-set similarity metric to construct hash items and learn binary encoding. Supervised cross-modal hashing uses manually annotated labels for effective compact representation learning to obtain information representation and compact encoding. After introducing annotation information, cross-modal hashing methods usually produce superior representation effects, which are better than unsupervised methods. In particular, as deep neural networks have achieved remarkable results in computer vision, natural language processing, and information retrieval, new deep cross-modal hashing methods have been proven to have stronger capabilities in practical retrieval applications by combining feature representation with discrete encoding learning through an end-to-end framework. By leveraging the unique representation capabilities of neural networks, deep cross-modal hashing can capture rich semantic information from samples of different modalities, thereby effectively narrowing the gap between modalities and achieving excellent retrieval performance.

[0003] Although existing deep cross-modal hashing algorithms have made significant progress, there are still some limitations. Generally, in multi-label retrieval scenarios, most existing supervised cross-modal hashing methods use fixed-level semantic affinity defined by manual labels as a supervisory signal to guide hashing learning. These methods usually map data to Euclidean space and use Euclidean distance to measure the correlation between data points. However, these methods are prone to the "curse of dimensionality" in high-dimensional space, which affects the performance of the model.

[0004] In recent years, hypersphere embedding technology has become mainstream in fields such as face and speech recognition. In the spherical embedding space, cosine similarity is often used to measure the similarity between vectors. However, when the embedding vector contains multiple categories, the label of each category will cause the instance's embedding vector to shift in different directions. If these directions cancel each other out, the embedding vector may be pulled back to the origin. In addition, existing methods usually ignore the modulus (L2 norm) of the vector, which can actually reflect the uncertainty or ambiguity of the instance. Ignoring this information may result in the loss of important confidence information. Summary of the invention

[0005] In order to optimize the image and text retrieval effect, the present invention provides an image and text cross-modal retrieval method, device and storage medium based on spherical hashing.

[0006] The first aspect of the present invention provides a cross-modal image-text retrieval method based on spherical hashing, which comprises the following steps:

[0007] Training a cross-modal image-text retrieval model, where the cross-modal image-text retrieval model includes an image feature encoder, a text feature encoder, and a hash layer;

[0008] Inputting the search text or the search image into the image-text cross-modal retrieval model, the image-text cross-modal retrieval model determines A first similarities between the text features of the search text and the image features of A images, or determines B second similarities between the image features of the search image and the text features of B texts;

[0009] For X first similarities greater than a preset similarity threshold among A first similarities, images corresponding to the X first similarities are determined as search results; or, for Y second similarities greater than a preset similarity threshold among B second similarities, texts corresponding to the Y second similarities are determined as search results; wherein A, B, X, and Y are all non-negative integers, A≥X, B≥Y;

[0010] The steps of training the image-text cross-modal retrieval model include:

[0011] Acquire sample data, wherein the sample data includes a sample image and a sample text;

[0012] Processing the sample image using the image feature encoder to obtain the sample image features;

[0013] Process the sample text using a text feature encoder to obtain sample text features;

[0014] A hash layer is used to map sample image features or sample text features into a hash code, wherein the parameters of the hash layer are updated according to the random spherical von Mises-Fisher loss as shown in the following formula;

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] in, represents the random spherical von Mises-Fisher loss, represents the random spherical von Mises-Fisher loss of the image modality, The random sphere von Mises-Fisher loss representing the text modality, represents the random spherical von Mises-Fisher loss between modes; [·] represents the expected value of the image modality class binary code, [·] represents the expected value of the text modality binary code, [·] represents the expected value of the binary code between the modes; C represents the number of categories; represents the learnable weight vector of the image modality; E[·] represents the expectation; A binary-like code representing the image modality, Represents the text mode binary code, Represents binary-like codes between modalities; Indicates the label, represents category weight; n represents a positive integer, and i represents a positive integer.

[0020] In this method, the spherical von Mises-Fisher distribution is introduced into cross-modal hashing to explore uncertainty information to guide the hashing process and generate discriminative modality-invariant hash codes. At the same time, in order to retain the semantic information of the sample's specific mode, a random spherical von Mises-Fisher loss function is proposed to achieve the alignment of different modes and semantic embeddings in the form of probability distribution.

[0021] In some optional embodiments, the image feature encoder is a Transformer encoder, and the text feature encoder is a Transformer encoder.

[0022] In some optional embodiments, the parameters of the hash layer are updated according to the overall loss as shown below;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] in, represents the overall loss, represents the random spherical von Mises-Fisher loss, represents the semantic similarity loss, represents the semantic similarity loss of the image modality, represents the semantic similarity loss of the text modality, Represents the semantic similarity loss between modalities; Indicates label similarity, represents the similarity between sample pairs in the image modality, Represents the similarity between samples in the text modality, Represents the similarity between sample pairs between modalities; i represents a positive integer, j represents a positive integer, and n represents a positive integer.

[0029] In some optional embodiments, the step of processing the sample image using the image feature encoder includes:

[0030] For the sample image, a convolution operation is used to generate patches, and each patch is assigned a corresponding position code, as shown in the following formula;

[0031] ;

[0032] in, Indicates a patch, represents a sample image, Represents the position code of the sample image;

[0033] The patch is input into the image feature encoder to generate a sample image feature vector.

[0034] In some optional embodiments, the step of processing the sample text using the text feature encoder includes:

[0035] For the sample text, the BPE method is used to encode each sample and the corresponding position code is added to each token, as shown in the following formula;

[0036] ;

[0037] in, represents the sample encoding with position encoding, Represents sample text, Represents the position code of the sample text;

[0038] Will Input into the text feature encoder to generate a sample text feature vector.

[0039] In some optional embodiments, the step of mapping the sample image features or sample text features into hash codes using a hash layer includes:

[0040] The binary-like code representation is learned during the training process using a continuous relaxation method as shown below;

[0041] ;

[0042] ;

[0043] Use the element conversion function to convert the binary-like code representation into the actual binary code, as shown in the following formula;

[0044] ;

[0045] ;

[0046] ;

[0047] in, A binary-like code representation representing the features of a sample image, The binary-like code representation of the sample text features; tanh(·) represents the continuous relaxation method; (·) represents the hash function applied to the image, (·) represents the hash function applied to the text; represents image samples; N represents the total number of samples, k represents the length of the generated hash code; sign(·) represents the sign function; A binary code representing the image modality, Binary code representing text mode.

[0048] In some optional embodiments, in the hashing layer, the binary code representation of the sample image features or the sample text features and the learnable weight vector of each category are projected onto the unit hypersphere by the following formula;

[0049]

[0050]

[0051]

[0052] in, represents the mean unit vector, represents the isotropic concentration parameter; (·) represents the probability density function of the vMF distribution, random dimensional unit vector The probability density function of the vMF distribution is calculated by the following formula;

[0053] ;

[0054] in, , , represents the normalization parameter defined by;

[0055] ;

[0056] in, represents the modified Bessel function of the first kind of order d / 2−1.

[0057] A second aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the above-mentioned spherical hash-based image and text cross-modal retrieval method according to the computer program.

[0058] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program; when the computer program is executed by a processor, the computer program implements the above-mentioned cross-modal image and text retrieval method based on spherical hashing.

[0059] The technical solution of the embodiment of the present invention has the following beneficial effects: in the hash layer, the semantic information of the sample-specific pattern is retained by jointly optimizing the random spherical von MisesFisher loss and the semantic similarity loss, thereby improving the retrieval performance of the image-text cross-modal retrieval model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Schematic diagram of the network structure of the image-text cross-modal retrieval model in an embodiment of the present invention.

[0061] Figure 2 PR curves and TopN-precision curves of different methods on the MS COCO dataset, where (a) is the PR curve of the image-to-text retrieval task, (b) is the PR curve of the text-to-image retrieval task, (c) is the TopN-precision curve of the image-to-text retrieval task, and (d) is the TopN-precision curve of the text-to-image retrieval task.

[0062] Figure 3 PR curves and TopN-precision curves of different methods on the MIRFLICKR-25K dataset, where (a) is the PR curve of the image-to-text retrieval task, (b) is the PR curve of the text-to-image retrieval task, (c) is the TopN-precision curve of the image-to-text retrieval task, and (d) is the TopN-precision curve of the text-to-image retrieval task.

[0063] Figure 4PR curves and TopN-precision curves of different methods on the IAPR TC-12 dataset, where (a) is the PR curve of the image-to-text retrieval task, (b) is the PR curve of the text-to-image retrieval task, (c) is the TopN-precision curve of the image-to-text retrieval task, and (d) is the TopN-precision curve of the text-to-image retrieval task.

[0064] Figure 5 PR curves and TopN-precision curves of different methods on the NUS-WIDE dataset, where (a) is the PR curve of the image-to-text retrieval task, (b) is the PR curve of the text-to-image retrieval task, (c) is the TopN-precision curve of the image-to-text retrieval task, and (d) is the TopN-precision curve of the text-to-image retrieval task.

[0065] Figure 6 P@H≤2 curves of different methods on the MS COCO dataset, where (a) is the P@H≤2 curve for the image-to-text retrieval task, and (b) is the P@H≤2 curve for the text-to-image retrieval task.

[0066] Figure 7 P@H≤2 curves of different methods on the MIRFLICKR-25K dataset, where (a) is the P@H≤2 curve for the image-to-text retrieval task, and (b) is the P@H≤2 curve for the text-to-image retrieval task.

[0067] Figure 8 P@H≤2 curves of different methods on the IAPR TC-12 dataset, where (a) is the P@H≤2 curve for the image-to-text retrieval task, and (b) is the P@H≤2 curve for the text-to-image retrieval task.

[0068] Fig. 9 P@H≤2 curves of different methods on the NUS-WIDE dataset, where (a) is the P@H≤2 curve for the image-to-text retrieval task, and (b) is the P@H≤2 curve for the text-to-image retrieval task. DETAILED DESCRIPTION

[0069] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] This paper proposes a cross-modal image-text retrieval method based on spherical hashing. In order to learn the explicit uncertainty between heterogeneous modalities and the learned embedding space, the spherical von Mises-Fisher distribution is first introduced into deep cross-modal hashing, where the direction of sample embedding controls its position on the hypersphere, thereby controlling its semantic content and the determinism of its norm parameterized distribution. In addition, the alignment of different modes and semantic embeddings is achieved based on random spherical von Mises-Fisher loss and semantic similarity loss.

[0071] The image-text cross-modal retrieval method based on spherical hashing is implemented by training an image-text cross-modal retrieval model. Figure 1 As shown, the framework of the image-text cross-modal retrieval model of an embodiment of the present invention mainly includes two parts: feature learning and hash learning. In terms of feature learning, two Transformer encoders are used to obtain the latent semantic representation of the image modality and the text modality respectively. In the hash learning module, the semantic information of the sample-specific pattern is retained by jointly optimizing the random spherical von MisesFisher loss and the semantic similarity loss.

[0072] The image-text cross-modal retrieval model includes an image feature encoder, which uses a Transformer encoder as the backbone to extract embedded representations of images and text. The image feature encoder consists of 12 encoder blocks, each of which contains a multi-head self-attention mechanism and a multi-layer perceptron (MLP). Specifically, for an image sample , a convolution operation is used to generate patches, each patch is assigned its corresponding position code , as shown below;

[0073] ;

[0074] Finally, these patches are fed into the image feature encoder to generate the image feature vector ,in is the image feature encoder, is the network parameter.

[0075] The image-text cross-modal retrieval model includes a text feature encoder. Similarly, the Transformer encoder is also used to extract text modal features. Specifically, for a given text sample, the BPE (Byte Pair Encoding) method is used to encode each sample and encode the corresponding position. Added to each token as shown below;

[0076] ;

[0077] Then, The input is sent to the text feature encoder, which consists of 12 encoder layers, each of which contains 8 multi-head attention mechanisms. Finally, the semantic features of the text , generated by the text feature encoder, where represents the text feature encoder, Represents the parameters of the text network.

[0078] The cross-modal image-text retrieval model includes a hash layer, which maps the extracted high-dimensional features into a k-bit hash code (k is the length of the generated hash code). In this discrete space, the Hamming distance between similar samples is significantly smaller than the Hamming distance between semantically unrelated samples.

[0079] and Represent the binary codes corresponding to the image and text modalities, N is the number of samples, and k represents the length of the generated hash code. Considering that binary optimization is considered to be an NP-hard problem, a continuous relaxation method (tanh) is used to learn the binary codes throughout the training process, as shown in the following formula;

[0080] ;

[0081] .

[0082] In the hash code generation phase, the binary-like representation is converted into the actual binary code using an element-wise conversion function (sign) as shown below;

[0083] ;

[0084] ;

[0085] .

[0086] Existing cross-modal hashing usually adopts deterministic embedding, ignoring the uncertainty measurement caused by hash projection. Therefore, the ability to retain inherent multimodal semantics is limited, thereby limiting the retrieval performance. At the same time, in the multi-label retrieval scenario, most of the existing supervised cross-modal hashing methods use fixed-level semantic affinity defined by manual labels as a supervisory signal to guide hash learning. These methods usually determine the category by mapping to Euclidean space and using Euclidean distance to measure the distance from the data point to the center of each category. However, this method suffers from the "curse of dimensionality" in high-dimensional space, which affects the performance of the model. Compared with existing methods, this embodiment establishes a random spherical von Mises-Fisher loss based on the spherical von Mises-Fisher distribution. The von Mises-Fisher (vMF) distribution is the maximum entropy distribution on the hypersphere and is generally considered to be a generalization of the multivariate normal distribution in the hypersphere domain. It is composed of the mean unit vector and isotropic concentration parameter Parameterized. Random dimensional unit vector The probability density function (PDF) of the vMF distribution is:

[0087] ;

[0088] in , , normalized parameter The definition is as follows:

[0089] ;

[0090] in is the modified Bessel function of the first kind of order d / 2−1. Similar to the Gaussian distribution, the vMF distribution is defined by two parameters: the mean unit vector and concentration parameters .parameter Reflects the average direction The degree of concentration of surrounding distribution. As increases, the distribution becomes more tightly concentrated in the mean direction.

[0091] Different from the Euclidean space projection, in the framework proposed in this embodiment, the binary representation of multimodal samples and each category , The learnable weight vector of can be projected onto the unit hypersphere by the following formula superior:

[0092]

[0093]

[0094]

[0095] Norm Directly control the distribution expansion, where the zero norm produces a uniform distribution on the surface of the hypersphere. For the image modality, the random spherical von Mises-Fisher loss can be expressed as follows:

[0096] ;

[0097] where C represents the number of categories. By applying Jensen’s inequality, we can deduce This allows marginalization , thus expressing it as relative to The expected value form is as shown below;

[0098] ;

[0099] in This goal can be achieved by To approximate it by sampling, N samples are enough, where N refers to randomly selecting N samples from all the samples.

[0100] During the testing phase, vMF used and Each Monte Carlo sample approximates .

[0101] For the text modality, similar to the image modality, the random sphere von Mises-Fisher loss for text is calculated as follows:

[0102] .

[0103] In the cross-modal retrieval scenario, how to jointly learn intra-modal and inter-modal interactions is a key factor in bridging the semantic and heterogeneity gap. From a multimodal perspective, the random sphere von Mises-Fisher loss of text is as follows:

[0104] .

[0105] Finally, from the intra-modal and inter-modal perspectives, the overall multimodal random spherical von Mises-Fisher loss is as follows:

[0106] .

[0107] In order to preserve modality-specific semantic information, a semantic similarity loss is introduced in cross-modal hashing. Specifically, for a given image semantic feature and text semantic features , calculate the relative similarity between samples, and then align them according to the label L to promote separation between classes. Specifically, the semantic similarity loss is calculated using cross entropy loss. For the image modality, it is as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] in Indicates label similarity, Represents the similarity between pairs of samples in the image modality.

[0112] For the text modality, similar to the image modality, the semantic similarity loss of text is calculated as follows:

[0113] ;

[0114] .

[0115] From a multimodal perspective, the multimodal semantic similarity loss is calculated as follows:

[0116] ;

[0117] .

[0118] The overall multimodal semantic similarity loss is as follows:

[0119] .

[0120] By jointly optimizing the random spherical von Mises-Fisher loss and the semantic similarity loss, the overall objective function of the framework designed in this embodiment can be expressed as:

[0121] .

[0122] The above-mentioned image-text cross-modal retrieval model is trained using sample data, wherein the sample data includes sample images and sample texts. The retrieval text or retrieval image is input into the trained image-text cross-modal retrieval model to realize image-text cross-modal retrieval. The image-text cross-modal retrieval model determines A first similarities between the text features of the retrieval text and the image features of A (for example, 1000) images, or determines B second similarities between the image features of the retrieval image and the text features of B (for example, 1000) texts. For X (for example, 100) first similarities greater than a preset similarity threshold among the A first similarities, the images corresponding to the X first similarities are determined as retrieval results; or, for Y (for example, 100) second similarities greater than a preset similarity threshold among the B second similarities, the texts corresponding to the Y second similarities are determined as retrieval results.

[0123] In order to demonstrate the performance of the cross-modal image-text retrieval method based on spherical hashing in the embodiment of the present invention, experimental verification was carried out using the MIRFLICKR-25K dataset, NUS-WIDE dataset, MS COCO dataset, and IAPR TC-12 dataset (four benchmark datasets). The experiment used the open source PyTorch library and was conducted on a Linux server equipped with an NVIDIA RTX 3090 GPU.

[0124] The MIRFLICKR-25K dataset contains a total of 24,581 pairs of images and text, each of which is from the Flickr website and contains the corresponding descriptive text. This comprehensive dataset covers 24 different semantic categories, ensuring that each category is represented by at least one instance covering all included categories.

[0125] The NUS-WIDE dataset is a prominent and multifaceted cross-modal resource that is frequently used in the field of deep hashing. To be consistent with other methods, a preparation step was performed, including eliminating categories with sparse sample representation. After this refinement process, the dataset has been reduced to contain a total of 195,834 image-text pairs covering 21 well-known categories.

[0126] The MS COCO dataset contains 82,785 images for training and 40,504 images for validation, each of which is paired with a corresponding text description and label. Importantly, each image-text pair is classified into one of the 80 unique categories present in the dataset.

[0127] The IAPR TC-12 dataset contains 20,000 image-text pairs, each of which consists of a static natural image and a corresponding text description. These image-text pairs are divided into 291 different groups, and each image-text pair belongs to at least one category.

[0128] Eleven typical deep cross-modal hashing methods were selected for comparison with the cross-modal image-text retrieval method based on spherical hashing (DSSH) of the embodiment of the present invention, including deep cross-modal hashing (DCMH), self-supervised adversarial hashing (SSAH), cross-modal Hamming hashing (CMHH), adversary-guided asymmetric hashing (AGAH), deep adversarial discrete hashing (DADH), self-constrained attention hashing network (SCAHN), multi-label enhanced self-supervised deep cross-modal hashing (MESDCH), distinguishable cross-modal hashing transformer (DCHMT), modality-invariant asymmetric network (MIAN), deep neighborhood-aware proxy hashing (DNPH) and RDPH. The comparison method is based on the official source code, and the relevant parameter settings refer to the original paper. It should be noted that in order to ensure the fairness of the experiment, all comparison methods (including DCMH, CMHH, AGAH, DADH and SCAHN) use the same backbone architecture to evaluate the retrieval performance of the model.

[0129] To evaluate the performance of the model in cross-modal retrieval, five commonly used evaluation metrics are used: mean average precision (mAP), precision-recall (PR) curve, TopN precision curve, precision with Hamming radius 2 (P@H≤2) curve, and normalized discounted cumulative gain with top 1000 returned samples (NDCG@1000). The mAP score represents the average accuracy of the model on the entire dataset and is a standard performance metric widely recognized in the field of information retrieval. The PR curve shows the relationship between recall and precision, which can provide insights into the overall retrieval performance. The TopN precision curve measures accuracy based on the returned Top-N samples. P@H≤2 measures the average accuracy within a Hamming radius of 2. NDCG@1000 is a metric that measures the relevance of search results to user queries, and its value increases when the returned ranked samples show higher similarity. In the experiment, the number of rankings is configured to be 1000.

[0130] First, the mAP of 16, 32, and 64 bits was calculated on four benchmark datasets, focusing on two retrieval tasks: image-to-text (I2T) and text-to-image (T2I) retrieval. The detailed results are shown in Table 1. It can be seen that by introducing VMF distribution in cross-modal hashing to capture the uncertainty in the embedding space, the DSSH of the embodiment of the present invention outperforms the existing state-of-the-art methods. Specifically, on the MIRFLICKR-25K dataset, the DSSH framework surpasses the best baseline method DNPH in image-to-text (I2T) and text-to-image (T2I) tasks at three different hash lengths, with improvements of 2.23% and 2.07%, respectively. On the NUSWIDE dataset, DSSH surpasses the state-of-the-art DNPH methods by 3.3% to 3.68% in image-to-text retrieval tasks and by 1.09% to 1.72% in text-to-image retrieval tasks. For the MS COCO dataset, DSSH achieves improvements of 5.33%, 3.89%, and 1.7% for image-to-text retrieval tasks at 16, 32, and 64-bit hash code lengths, respectively, and 3.61%, 1.94%, and 0.11% for text-to-image retrieval tasks. Compared to the best performing RDPH method on the IAPRTC-12 dataset, DSSH achieves significant improvements of 18.87% and 14.72%, respectively.

[0131] Table 1. mAP of different methods on four benchmark datasets with 16-bit, 32-bit and 64-bit

[0132]

[0133] Secondly, NDCG@1000 is calculated on four benchmark datasets. The detailed results are shown in Table 2.

[0134] Table 2 NDCG@1000 of different methods on four benchmark datasets

[0135]

[0136] In order to further evaluate the effectiveness of the DSSH of the embodiment of the present invention, PR curves and TopN-precision curves are plotted on four public data sets using 32-bit binary encoding, as shown in Figure 2-Figure 5 In addition, the P@H≤2 curve is shown in Figure 6-Figure 9 Compared with other comparative methods, the DSSH method of the embodiment of the present invention achieves good retrieval performance in terms of PR curve, TopN-precision curve and P@H≤2.

[0137] The above is only a preferred embodiment of the present invention, and is not any formal or substantial limitation of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the method of the present invention, and these improvements and supplements should also be regarded as the protection scope of the present invention. Any technician familiar with this profession, without departing from the spirit and scope of the present invention, can make some changes, modifications and evolutions of the technical content disclosed above, which are equivalent embodiments of the present invention; at the same time, any changes, modifications and evolutions of any equivalent changes made to the above embodiments based on the essential technology of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A cross-modal image and text retrieval method based on spherical hashing, characterized in that: The following steps are involved: Training a cross-modal image-text retrieval model, where the cross-modal image-text retrieval model includes an image feature encoder, a text feature encoder, and a hash layer; Inputting the search text or the search image into the image-text cross-modal retrieval model, the image-text cross-modal retrieval model determines A first similarities between the text features of the search text and the image features of A images, or determines B second similarities between the image features of the search image and the text features of B texts; For X first similarities greater than a preset similarity threshold among A first similarities, images corresponding to the X first similarities are determined as search results; or, for Y second similarities greater than a preset similarity threshold among B second similarities, texts corresponding to the Y second similarities are determined as search results; wherein A, B, X, and Y are all non-negative integers, A≥X, B≥Y; The steps of training the image-text cross-modal retrieval model include: Acquire sample data, wherein the sample data includes a sample image and a sample text; Processing the sample image using the image feature encoder to obtain the sample image features; Process the sample text using a text feature encoder to obtain sample text features; A hash layer is used to map sample image features or sample text features into a hash code, wherein the parameters of the hash layer are updated according to the random spherical von Mises-Fisher loss as shown in the following formula; ； ； ； ； in, represents the random spherical von Mises-Fisher loss, represents the random spherical vonMises-Fisher loss of the image modality, The random sphere von Mises-Fisher loss representing the text modality, represents the random spherical von Mises-Fisher loss between modes; [·] represents the expected value of the image modality class binary code, [·] represents the expected value of the text modality binary code, [·] represents the expected value of the binary code between the modes; C represents the number of categories; represents the learnable weight vector of the image modality; E[·] represents the expectation; A binary-like code representing the image modality, Represents the text mode binary code, Represents binary-like codes between modalities; Indicates the label, represents category weight; n represents a positive integer, and i represents a positive integer.

2. The method according to claim 1, characterized in that: The image feature encoder is a Transformer encoder, and the text feature encoder is a Transformer encoder.

3. The method according to claim 1, characterized in that: Update the parameters of the hashing layer according to the overall loss as shown below; ； ； ； ； ； in, represents the overall loss, represents the random spherical von Mises-Fisher loss, represents the semantic similarity loss, represents the semantic similarity loss of the image modality, represents the semantic similarity loss of the text modality, Represents the semantic similarity loss between modalities; Indicates label similarity, represents the similarity between sample pairs in the image modality, Represents the similarity between samples in the text modality, Represents the similarity between sample pairs between modalities; i represents a positive integer, j represents a positive integer, and n represents a positive integer.

4. The method according to claim 3, characterized in that The steps of processing a sample image using an image feature encoder include: For the sample image, a convolution operation is used to generate patches, and each patch is assigned a corresponding position code, as shown in the following formula; ； in, Indicates a patch, represents a sample image, Represents the position code of the sample image; The patch is input into the image feature encoder to generate a sample image feature vector.

5. The method according to claim 4, characterized in that The steps of processing sample text using the text feature encoder include: For the sample text, each sample is encoded using the BPE method, and the corresponding positional encoding is added to each token, as shown in the following formula; ； in, represents the sample encoding with position encoding, Represents sample text, Represents the position code of the sample text; Will Input into the text feature encoder to generate a sample text feature vector.

6. The method according to claim 5, characterized in that The steps of mapping sample image features or sample text features into hash codes using a hash layer include: The binary-like code representation is learned during the training process using a continuous relaxation method as shown below; ； ； Use the element conversion function to convert the binary-like code representation into the actual binary code, as shown in the following formula; ； ； ； in, A binary code representation of sample image features, The binary-like code representation of the sample text features; tanh(·) represents the continuous relaxation method; (·) represents the hash function applied to the image, (·) represents the hash function applied to the text; represents image samples; N represents the total number of samples, k represents the length of the generated hash code; sign(·) represents the sign function; A binary code representing the image modality, Binary code representing text mode.

7. The method according to claim 6, characterized in that In the hashing layer, the binary code representation of the sample image features or sample text features and the learnable weight vector of each category are projected onto the unit hypersphere by the following formula; ； ； ； in, represents the mean unit vector, represents the isotropic concentration parameter; (·) represents the probability density function of the vMF distribution, random dimensional unit vector The probability density function of the vMF distribution is calculated by the following formula; ； in, , , represents the normalization parameter defined by; ； in, represents the modified Bessel function of the first kind of order d / 2−1.

8. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the image-text cross-modal retrieval method based on spherical hashing as described in any one of claims 1 to 7 according to the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program; when the computer program is executed by a processor, it implements the image-text cross-modal retrieval method based on spherical hashing as described in any one of claims 1 to 7.

Images