Cross-modal retrieval model training method and cross-modal retrieval method

Abstract

The invention relates to the technical field of cross-modal retrieval, and discloses a cross-modal retrieval model training method and a cross-modal retrieval method.The training method comprises the steps that a cross-modal retrieval model is constructed; training the cross-modal retrieval model by adopting a low-parameter training method; integrating a Hash layer in the cross-modal retrieval model to learn Hash mapping; constructing a sorting order among training samples of the sorting perception loss optimization input cross-modal retrieval model; the retrieval method comprises the following steps: inputting a query sample into a cross-modal retrieval model; performing feature extraction on a query sample by the cross-modal retrieval model; the cross-modal retrieval model converts the extracted high-dimensional features into low-dimensional and continuous binary-like codes; the binary-like codes are processed and converted into binary codes; comparing the similarity between the binary code of the query sample and the binary code of a storage sample in a database; and outputting a retrieval result semantically related to the query sample. The method is high in retrieval efficiency and good in retrieval performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal retrieval, and in particular relates to a training method for a cross-modal retrieval model and a cross-modal retrieval method. Background Art

[0002] With the rapid development of social media networks, multimedia data has exploded. In real life, the same entity can be described by data in multiple modalities, such as images, texts, etc. These heterogeneous data have the characteristics of heterogeneous underlying features and high-level semantic correlation. Therefore, how to effectively alleviate the semantic gap between different modal data and accurately retrieve the required information from heterogeneous data has become an important and urgent research topic. The traditional exact nearest neighbor retrieval scheme cannot meet the needs of fast and accurate retrieval due to the limitations of storage space and retrieval efficiency. Therefore, an approximate nearest neighbor retrieval scheme is proposed to improve retrieval efficiency.

[0003] Although existing deep hashing, as a representative approximate nearest neighbor algorithm, has performed well in the field of cross-modal image and text retrieval, there are still several problems: (1) The core goal of deep cross-modal hashing is to learn effective global ranking relations. However, existing methods usually use pairwise loss or triplet loss as alternatives to maintain the ranking objective. Pairwise loss involves calculating the relative distance between data pairs, while triplet loss considers the relative similarity between anchor points, positive samples, and negative samples. By minimizing the relative distance between samples, these counter-intuitive methods are indirectly related to ranking learning, resulting in the failure to capture global ranking relations. (2) Existing deep cross-modal hashing frameworks based on large visual language models usually use parameter full fine-tuning strategies for model optimization. However, parameter full fine-tuning strategies are expensive and inefficient, and require storing and deploying different model parameters for different datasets and hash code lengths.

[0004] In view of this, how to design a retrieval technology that satisfies both retrieval accuracy and retrieval efficiency is a technical problem to be solved by the present invention. Summary of the invention

[0005] The present invention provides a cross-modal retrieval model training method and a cross-modal retrieval method to achieve high-precision and efficient cross-modal retrieval.

[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0007] In one aspect, the present invention provides a method for training a cross-modal retrieval model, comprising the following steps:

[0008] Build a cross-modal retrieval model;

[0009] Using a low-parameter training method to train the cross-modal retrieval model;

[0010] Integrating a hash layer in the cross-modal retrieval model to learn a hash mapping;

[0011] A rank-aware loss is constructed to optimize the ranking order between the training samples input into the cross-modal retrieval model.

[0012] As described above, the cross-modal retrieval model training method adopts a CLIP-based deep hashing model, and the CLIP-based deep hashing model includes a plurality of Transformer encoders and hashing layers of different modalities.

[0013] The training method of the cross-modal retrieval model as described above, the steps of using the low-parameter training method to train the cross-modal retrieval model include:

[0014] Freeze the pre-trained parameters of the CLIP model, set learnable parameters in each layer of the Transformer encoder, the learnable parameters accept gradient updates, and perform parameter fine-tuning through back propagation. The input after setting the learnable parameters in each layer of the Transformer encoder is as follows:

[0015] E i =T i ([P i-1 , E i-1 ])

[0016] Among them, E i Represents the input of the Transformer encoder at the i-th layer; E i-1 represents the input of the Transformer encoder at the i-1th layer; T i represents the i-th layer of the Transformer encoder; P i-1 Represents the learnable parameters set in the Transformer encoder.

[0017] In the training method of the cross-modal retrieval model as described above, the step of integrating a hash layer to learn a hash mapping in the cross-modal retrieval model includes:

[0018] An activation function is set in the hash layer so that the hash layer outputs continuous binary-like codes. The output of the hash layer is as follows:

[0019] H = tanh(W(F)+O)

[0020] Wherein, G represents the output of the hash layer; tanh(·) represents the activation function; and o represents the bias parameter.

[0021] The training method of the cross-modal retrieval model described above constructs the objective function of the ranking-aware loss as follows:

[0022]

[0023] Among them, L ranking represents the objective function of rank-aware loss; α is the first hyperparameter used to balance the intra-modal and inter-modal rank-aware losses; β is the second hyperparameter used to balance the intra-modal and inter-modal rank-aware losses; Represents the ranking-aware loss function from the first modality to the second modality; represents the ranking-aware loss function of the second modality crossing to the first modality; represents the first modality intra-ranking aware loss function; represents the second intra-modality ranking-aware loss function.

[0024] In the training method of the cross-modal retrieval model described above, the first modality intra-ranking perception loss function and the second modality intra-ranking perception loss function are respectively as follows:

[0025]

[0026] in, represents the first modality intra-ranking perception loss function; n represents the number of samples; represents the average precision of retrieving other images using the training images as queries; represents the second modality intra-ranking aware loss function; represents the average precision of retrieving other texts using the training text as query;

[0027] The ranking-aware loss function of the first modality to the second modality and the ranking-aware loss function of the second modality to the first modality are respectively as follows:

[0028]

[0029] in, represents the ranking-aware loss function from the first modality to the second modality; n represents the number of samples; represents the average precision of retrieving other texts using training images as queries;

[0030] represents the ranking-aware loss function of the second modality crossing to the first modality; Represents the average precision of retrieving other images using the training text as queries.

[0031] In one aspect, the present invention provides a cross-modal retrieval method, comprising the following steps:

[0032] Inputting the query sample into the cross-modal retrieval model trained as described in any of the above items;

[0033] The cross-modal retrieval model performs feature extraction on the query sample;

[0034] The cross-modal retrieval model converts the extracted high-dimensional features into low-dimensional and continuous binary-like codes;

[0035] Processing the binary-like code and converting it into a binary code;

[0036] Comparing the similarity between the binary code of the query sample and the binary code of the samples stored in the database;

[0037] Output the search results that are semantically related to the query sample.

[0038] As described above, the cross-modal retrieval method, wherein the cross-modal retrieval model converts the extracted high-dimensional features into low-dimensional and continuous binary-like codes, includes:

[0039] The hash layer maps the extracted high-dimensional features into k-bit hash codes, which are low-dimensional and continuous binary-like codes.

[0040] In the cross-modal retrieval method described above, the process of converting the binary-like code into a binary code is as follows:

[0041] B=sign(H)

[0042] Wherein, B represents the converted binary code; sign(·) represents the sign function, if ·>1, sign(·)=1; otherwise sign(·)=-1; H represents the binary-like code.

[0043] In the cross-modal retrieval method as described above, the binary encoding of the query sample and the binary encoding of the sample stored in the database are compared for similarity by calculating the Hamming distance. The smaller the Hamming distance, the more semantically relevant the query sample is to the stored sample.

[0044] Compared with the prior art, the advantages and positive effects of the present invention are:

[0045] (1) When training the cross-modal retrieval model, a low-parameter training method is used to train the model, thereby optimizing the model parameters, improving the training and deployment efficiency of the cross-modal retrieval model, and effectively reducing the number of trainable parameters that need to be adjusted, so that the model can maintain its learning ability while saving computing resources and improving training efficiency; a new ranking-aware loss is constructed to effectively capture high-quality global ranking relationships between cross-modal data;

[0046] (2) When using the cross-modal retrieval model for retrieval, the retrieval accuracy is high, the retrieval efficiency is fast, and the robustness is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0048] Figure 1 A flowchart of an embodiment of a method for training a cross-modal retrieval model of the present invention;

[0049] Figure 2 It is a structural schematic diagram of an embodiment of a cross-modal retrieval model of the present invention;

[0050] Figure 3 This is a flowchart of an embodiment of a cross-modal retrieval method of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] It should be noted that in the description of the present invention, the terms "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings, which are only for the convenience of description, and do not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0053] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may include that the first and second features are in direct contact, or may include that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, a first feature being "above", "above" and "above" a second feature includes that the first feature is directly above and obliquely above the second feature, or simply indicates that the first feature is higher in level than the second feature. A first feature being "below", "below" and "below" a second feature includes that the first feature is directly below and obliquely below the second feature, or simply indicates that the first feature is lower in level than the second feature.

[0055] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplicity and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed. In addition, the present invention provides various specific examples of processes and materials, but those of ordinary skill in the art can be aware of the application of other processes and / or the use of other materials.

[0056] like Figure 1 to Figure 2 As shown, the present application provides a training method for a cross-modal retrieval model, which constructs a new ranking-aware loss to effectively capture high-quality global ranking relationships between heterogeneous data; on this basis, a low-parameter training scheme is used to optimize the model parameters, which significantly reduces the amount of trainable parameters of the model, thereby improving the efficiency of the cross-modal retrieval model while ensuring performance standards.

[0057] A method for training a cross-modal retrieval model comprises the following steps:

[0058] Construct a cross-modal retrieval model. Specifically,

[0059] The cross-modal retrieval model adopts a deep hashing model based on CLIP (Contrastive Language-Image Pre-Training). The CLIP model includes multiple Transformer encoders of different modalities.

[0060] Each layer of the Transformer encoder is a structure that processes data serially, organized by Multi-head Self Attention (MSA), Multi-layer Perception (MLP), and residual connections. Taking the input image data as an example, the input image data is first processed in parallel by multi-head self-attention to capture the long visual dependencies and semantic information within the image; then, the multi-head self-attention is sent to a multi-layer perceptron for nonlinear transformation and mapping of image features; finally, residual connections are applied between the input and output of the multi-head self-attention and between the input and output of the multi-layer perceptron to ensure the stability of gradient propagation. When the input data is text data, the processing flow is the same as that of the input data is image data, so it will not be repeated here.

[0061] In this embodiment, taking images and texts as examples, the CLIP-based deep hash model includes an image Transformer encoder, a text Transformer encoder, and a hash layer.

[0062] For the specific image x of the input contrast multimodal pre-trained model i First, through the convolution operation and flattening operation, x i Divide into m fixed-size image blocks, denoted as These image patches make up the complete image. As input to the image Transformer encoder.

[0063] For the specific text y of the input comparison multimodal pre-trained model i First, the Byte Pair Encoding (BPE) algorithm is used to convert the text into m word vectors, recorded as These word vectors make up the complete text As input to the text Transformer encoder.

[0064] The cross-modal retrieval model is trained using a low-parameter training method. Specifically:

[0065] This application optimizes the parameters of the cross-modal retrieval model through a low-parameter training strategy. The core idea of ​​the low-parameter training strategy is to load the pre-trained contrast multimodal pre-trained model in the initialization phase and freeze all its pre-trained parameters so that all pre-trained parameters do not accept any gradient updates in the training phase; then, additional learnable parameters are set in each layer of the Transformer encoder and initialized. These additional learnable parameters accept gradient updates in the training phase and fine-tune the parameters through back propagation. The input after setting the learnable parameters in each layer of the Transformer encoder is as follows:

[0066] E i =T i ([P i-1 , E i-1 ])

[0067] Among them, E i represents the input of the i-th layer Transformer encoder; E i-1 represents the input of the i-1th layer Transformer encoder; T i represents the i-th layer of the Transformer encoder; P i-1 Represents the learnable parameters set in the Transformer encoder.

[0068] For the image Transformer encoder, an additional set of learnable parameters is introduced in the input space of each Transformer encoder layer. The specific formula is as follows:

[0069]

[0070] in, Represents the input of the i-th layer image Transformer encoder; Represents the input of the i-1th layer image Transformer encoder; represents the i-th layer of the image Transformer encoder; Represents the learnable parameters set in the image Transformer encoder.

[0071] For the text Transformer encoder, an additional set of learnable parameters is introduced in the input space of each Transformer encoder layer. The specific formula is as follows:

[0072]

[0073] in, Represents the input of the i-th layer text Transformer encoder; Represents the input of the i-1th layer text Transformer encoder; represents the i-th layer of the text Transformer encoder; Represents the learnable parameters set in the text Transformer encoder.

[0074] During the training process of the cross-modal retrieval model, only these additional learnable parameters participate in the gradient update, while the pre-trained parameters remain frozen and do not undergo any update operations. This strategy of freezing pre-trained parameters effectively utilizes the rich knowledge of the pre-trained model and only updates relatively few learnable parameters, thereby significantly reducing the amount of trainable parameters. At the same time, by fine-tuning the additional learnable parameters, the cross-modal retrieval model can be fine-tuned according to the characteristics of the cross-modal data, thereby improving the performance of the cross-modal retrieval model on retrieval tasks.

[0075] Integrate the hash layer to learn hash mapping in the cross-modal retrieval model. Specifically:

[0076] Since using binary coding to calculate the loss for back propagation in model training will make gradient calculation difficult, making it difficult for the model to converge, to overcome this problem, this application adopts a continuous relaxation strategy, specifically:

[0077] By setting the activation function in the hash layer, the hash layer outputs continuous binary-like codes. The output of the hash layer is as follows:

[0078] H = tanh(W(F)+O)

[0079] Wherein, H represents the output of the hash layer; tanh(·) represents the activation function; and o represents the bias parameter.

[0080] A rank-aware loss is constructed to optimize the ranking order between training samples input into the cross-modal retrieval model.

[0081] The ultimate goal of cross-modal retrieval models is to learn good ranking relations. However, previous retrieval models usually use pairwise loss or triplet loss as an alternative to ranking-based similarity preservation. These counter-intuitive methods are indirectly related to ranking learning, which makes it impossible to directly capture complex global ranking relations, thus limiting the performance of the model's cross-modal retrieval. In this application, by designing a differentiable average precision approximation as the overall optimization goal, a rank-aware loss is designed to learn the ranking similarity relationship between training samples to improve the performance of cross-modal similarity retrieval.

[0082] Average Precision (AP) is a commonly used evaluation metric in cross-modal retrieval. For example, given an image query By calculating the similarity between the image query and the text data, a similarity matrix can be obtained in is the positive sample set, is the negative sample set. For image query The calculation formula for average precision is as follows:

[0083]

[0084] in, represents the average precision of retrieving other texts using training images as queries; represents the positive sample set; Represents text y i Ranking position in the positive sample set; Represents text y i The ranked position in the search set.

[0085] Define sample y i The sorting relation R is as follows:

[0086]

[0087] in, is an indicator function. In order to facilitate calculation, a difference matrix is ​​introduced. The difference matrix D xy As shown below:

[0088]

[0089] in, Indicates the similarity between image and text samples.

[0090] Therefore, the calculation formula for average precision is as follows:

[0091]

[0092] in, is a Heaviside step function, whose derivative is a Dirac delta function, and the gradient of the Dirac delta function is either zero or discrete. Therefore, it is not feasible to directly apply average precision to optimize the cross-modal retrieval model, so the sigmoid function sig(i, τ) is used to replace Where τ is the temperature parameter, and the calculation formula of sig(i, τ) is as follows:

[0093]

[0094] Here, i represents the input of the function.

[0095] Therefore, for a specific image query Its differentiable average precision approximation is as follows:

[0096]

[0097] Then, the ranking-aware loss function from the first modality to the second modality is as follows:

[0098]

[0099] This is the image-text ranking perceptual loss, where represents the ranking-aware loss function from the first modality to the second modality; n represents the number of training samples. Represents the average precision of retrieving other texts using the training images as queries.

[0100] Similarly, for text queries The differentiable average precision approximation is as follows:

[0101]

[0102] Then, the ranking-aware loss function from the second modality to the first modality is as follows:

[0103]

[0104] This is the text-image ranking perceptual loss, where n represents the number of training samples; represents the ranking-aware loss function of the second modality crossing to the first modality; Represents the average precision of retrieving other images using the training text as queries.

[0105] In addition, the first intra-modal ranking-aware loss function is as follows:

[0106]

[0107] The second modality intra-ranking aware loss function is as follows:

[0108]

[0109] in, represents the first modality intra-ranking perception loss function; n represents the number of training samples; represents the average precision of retrieving other images using the training images as queries; represents the second modality intra-ranking aware loss function; Represents the average precision of retrieving other texts using the training text as query.

[0110] In summary, by jointly exploring the semantic similarity relationship between intra-modality and inter-modality, the objective function of the ranking-aware loss is as follows:

[0111]

[0112] Among them, L ranking represents the objective function of rank-aware loss; α is the first hyperparameter used to balance the intra-modal and inter-modal rank-aware losses; β is the second hyperparameter used to balance the intra-modal and inter-modal rank-aware losses; Represents the rank-aware loss function from the first modality to the second modality. represents the ranking-aware loss function of the second modality crossing to the first modality; represents the first modality intra-ranking aware loss function; represents the second intra-modality ranking-aware loss function.

[0113] The use of rank-aware loss to capture the global ranking relationship between heterogeneous data is achieved by comprehensively considering the position of training samples in the ranked list and the semantic similarity between them. The rank-aware loss can optimize the ranking order between samples so that related samples are closer to their correct positions, thereby learning a high-quality discrete space. The cross-modal retrieval model is trained so that similar samples in the Hamming space should be close to each other, while dissimilar samples should be far away. Therefore, through the process of optimizing the rank-aware loss, the cross-modal retrieval model can effectively learn how to accurately represent cross-modal data in the Hamming space, thereby achieving more accurate and efficient cross-modal similarity retrieval tasks.

[0114] like Figure 3 As shown, the present application provides a cross-modal retrieval method, comprising the following steps:

[0115] First, the query sample is input into the cross-modal retrieval model trained as described above.

[0116] Secondly, the cross-modal retrieval model extracts features from query samples.

[0117] Specifically, after the query sample is input into the cross-modal retrieval model, the Transformer encoder performs feature extraction on the query sample to extract high-dimensional features.

[0118] Then, the cross-modal retrieval model transforms the extracted high-dimensional features into low-dimensional and continuous binary-like encodings.

[0119] Specifically, the hash layer in the cross-modal retrieval model maps the high-dimensional features extracted by the Transformer encoder into a k-bit hash code, where k represents the length of the generated hash code, and the k-bit hash code is a low-dimensional and continuous binary-like code. A hash map is set for image and text data respectively, and the output of the hash layer is represented as follows with reference to the cross-modal retrieval model training method mentioned above:

[0120] H x =tanh(W x (F x )+O x )

[0121] H y =tanh(W y (F y )+O y )

[0122] Among them, H x represents the output of the hash layer when the query sample is an image; tanh(·) represents the activation function; W x Represents the weight parameter of the hash mapping of image data; F x Represents the high-dimensional features of the image; x represents the bias parameter of the hash map of image data; H y W represents the output of the hash layer when the query sample is text; y Represents the weight parameter of the hash mapping of text data; F y Represents the high-dimensional features of the text; y A paranoid parameter representing a hash map of text data.

[0123] Then, the binary-like code is processed and converted into binary code.

[0124] In the above steps, the hash layer outputs continuous binary-like codes, while discrete binary codes are required for subsequent similarity comparison. Therefore, it is necessary to use a bit-based conversion function to process the continuous binary-like codes. The formula is as follows:

[0125] B=sign(H)

[0126] Wherein, B represents the converted binary code; sign(·) represents the sign function, if ·>1, sign(·)=1; otherwise sign(·)=-1; H represents the binary-like code.

[0127] Compare the similarity between the binary encoding of the query sample and the binary encoding of the samples stored in the database.

[0128] Specifically, the binary encoding of the query sample and the binary encoding of the sample stored in the database are compared for similarity by calculating the Hamming distance, that is, comparing the number of bit differences between the two binary codes, which is usually calculated using an XOR operation. The smaller the Hamming distance, the more semantically relevant the query sample is to the stored sample.

[0129] Finally, the retrieval results that are semantically related to the query sample are output, that is, the data whose Hamming distance is close to the binary encoding distance of the query sample.

[0130] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0131] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.

Claims

1. A method for training a cross-modal retrieval model, characterized in that: The following steps are involved: Build a cross-modal retrieval model; Using a low-parameter training method to train the cross-modal retrieval model; Integrating a hash layer in the cross-modal retrieval model to learn a hash mapping; A rank-aware loss is constructed to optimize the ranking order between the training samples input into the cross-modal retrieval model.

2. The method for training a cross-modal retrieval model according to claim 1, characterized in that: The cross-modal retrieval model adopts a CLIP-based deep hashing model, and the CLIP-based deep hashing model includes Transformer encoders and hashing layers of multiple different modalities.

3. The method for training a cross-modal retrieval model according to claim 2, characterized in that: The step of training the cross-modal retrieval model using the low-parameter training method includes: Freeze the pre-trained parameters of the CLIP model, set learnable parameters in each layer of the Transformer encoder, the learnable parameters accept gradient updates, and perform parameter fine-tuning through back propagation. The input after setting the learnable parameters in each layer of the Transformer encoder is as follows: E i =T i ([P i-1 ,E i-1 ]) Among them, E i Represents the input of the Transformer encoder at the i-th layer; E i-1 represents the input of the Transformer encoder at the i-1th layer; T i represents the i-th layer of the Transformer encoder; P i-1 Represents the learnable parameters set in the Transformer encoder.

4. The method for training a cross-modal retrieval model according to claim 1, characterized in that: The steps of integrating a hash layer to learn a hash mapping in the cross-modal retrieval model include: An activation function is set in the hash layer so that the hash layer outputs continuous binary-like codes. The output of the hash layer is as follows: H=tanh(W(F)+O) wherein H represents the output of the hash layer; tanh(·) represents the activation function; and O represents the bias parameter.

5. The method for training a cross-modal retrieval model according to claim 1, characterized in that: The objective function for constructing the ranking-aware loss is as follows: Among them, I ranking represents the objective function of rank-aware loss; α is the first hyperparameter used to balance the intra-modal and inter-modal rank-aware losses; β is the second hyperparameter used to balance the intra-modal and inter-modal rank-aware losses; Represents the ranking-aware loss function from the first modality to the second modality; represents the ranking-aware loss function of the second modality crossing to the first modality; represents the first modality intra-ranking aware loss function; represents the second intra-modality ranking-aware loss function.

6. The method for training a cross-modal retrieval model according to claim 5, characterized in that: The first intra-modality ranking-aware loss function and the second intra-modality ranking-aware loss function are respectively as follows: in, represents the first modality intra-ranking perception loss function; n represents the number of samples; represents the average precision of retrieving other images using the training images as queries; represents the second modality intra-ranking aware loss function; represents the average precision of retrieving other texts using the training text as query; The ranking-aware loss function of the first modality to the second modality and the ranking-aware loss function of the second modality to the first modality are respectively as follows: in, represents the ranking-aware loss function from the first modality to the second modality; n represents the number of samples; represents the average precision of retrieving other texts using training images as queries; represents the ranking-aware loss function of the second modality crossing to the first modality; Represents the average precision of retrieving other images using the training text as queries.

7. A cross-modal retrieval method, characterized in that: The following steps are involved: Inputting the query sample into the cross-modal retrieval model trained as described in any one of claims 1 to 6; The cross-modal retrieval model performs feature extraction on the query sample; The cross-modal retrieval model converts the extracted high-dimensional features into low-dimensional and continuous binary-like codes; Processing the binary-like code and converting it into a binary code; Comparing the similarity between the binary code of the query sample and the binary code of the samples stored in the database; Output the search results that are semantically related to the query sample.

8. The cross-modal retrieval method according to claim 7, characterized in that: The cross-modal retrieval model converts the extracted high-dimensional features into low-dimensional and continuous binary-like codes including: The hash layer maps the extracted high-dimensional features into k-bit hash codes, which are low-dimensional and continuous binary-like codes.

9. The cross-modal retrieval method according to claim 7, characterized in that: The process of converting the binary code into binary code is as follows: B=sign(H) Wherein, B represents the converted binary code; sign(·) represents the sign function, if ·>1, sign(·)=1; otherwise sign(·)=-1; H represents the binary-like code.

10. The cross-modal retrieval method according to claim 9, characterized in that: The binary code of the query sample and the binary code of the sample stored in the database are compared for similarity by calculating the Hamming distance. The smaller the Hamming distance is, the more semantically relevant the query sample is to the stored sample.

Images