Image-text cross-modal hash code extraction model establishment method and application thereof

Abstract

The invention discloses an image-text cross-modal Hash code extraction model establishment method and application thereof, and belongs to the field of cross-modal retrieval, and the method comprises the following steps: respectively extracting Hash codes of images and Hash codes of texts in a training set and a current training batch by using a Hash model, and establishing an image global graph, an image local graph, a text global graph and a text local graph, reconstructing text features by using the image local graph, and reconstructing image features by using the text global graph; establishing a loss function, and performing model training until the loss function converges; the loss function comprises intra-modal comparison loss which is used for shortening the distance between adjacent Hash codes in the global graph and shortening the distance between non-adjacent Hash codes; the inter-modal comparison loss comprises similarity loss between each node in the global graph and another modal Hash code matched with the adjacent node; and reconstructing the loss, including the loss between the reconstructed feature and the original feature. According to the method, the accuracy of image-text cross-modal retrieval can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal retrieval, and more specifically, to a method for establishing an image-text cross-modal hash code extraction model and its application. Background Art

[0002] Efficient cross-modal retrieval of image and text data refers to retrieving similar texts based on images, or retrieving similar images based on texts. It has important applications in multimedia retrieval and processing. Cross-modal hashing methods can compress high-dimensional features into binary codes, reduce memory usage and improve retrieval efficiency. Among them, unsupervised cross-modal hashing methods do not require annotation of large-scale data sets and have become the mainstream solution in the field of cross-modal retrieval.

[0003] However, existing cross-image-text modality unsupervised hashing methods still have the following limitations:

[0004] (1) Insufficient modality alignment: Existing methods aim to improve the accuracy of the supervision signal but cannot effectively bridge the gap between modalities, resulting in suboptimal performance;

[0005] (2) Incomplete utilization of structural information: Existing methods only use implicit graph structures to establish more accurate pairwise similarities, ignoring the global and local information of the graph, which in turn affects the accuracy of cross-modal retrieval.

[0006] In general, the existing hashing methods for image-text cross-modal retrieval need to be further improved in terms of the accuracy of cross-modal retrieval. Summary of the invention

[0007] In view of the defects and improvement needs of the prior art, the present invention provides a method for establishing an image-text cross-modal hash code extraction model and its application, the purpose of which is to make full use of the global and local information of the graph and realize modal alignment during the training process of the hash code extraction model, so that hash codes from different modalities with similar semantics can be closely mapped together in the hash codes extracted by the hash code extraction model, and enhance the discriminability of the hash codes, thereby improving the accuracy of image-text cross-modal retrieval.

[0008] To achieve the above object, according to one aspect of the present invention, a method for establishing an image-text cross-modal hash code extraction model is provided, comprising:

[0009] (S1) using a first hash model to extract hash codes of images in a training set and a current training batch, and using a second hash model to extract hash codes of texts in the training set and the current training batch; the training data in the training set and the current training batch include matching images and texts; the first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract a text encoder T of text features and convert them into continuous hash codes;

[0010] (S2) establishing a graph structure according to the image hash code in the training set, the image hash code in the current training batch, the text hash code in the training set, and the text hash code in the current training batch, respectively, to obtain an image global graph, an image local graph, a text global graph, and a text local graph; the graph structure uses hash codes as nodes, and there are edges between each node and the K nodes with the highest similarity to it, and the weight of the edge is the similarity between the nodes; the similarity between the nodes is the similarity between the corresponding hash codes; K is a positive integer;

[0011] (S3) Reconstructing the local image into features using the first reconstruction model Use the second reconstruction model to reconstruct the text local map into features

[0012] (S4) establishing a loss function, and training the first hash model, the second hash model, the first reconstruction model, and the second reconstruction model according to the loss function; the loss function includes intra-modality contrast loss, inter-modality contrast loss, and reconstruction loss; the intra-modality contrast loss is used to shorten the distance between adjacent hash codes in the image global graph and the text global graph, and to lengthen the distance between non-adjacent hash codes in the image global graph and the text global graph; the inter-modality contrast loss includes the similarity loss between the text hash codes matched by each node in the image global graph and its adjacent nodes, and the similarity loss between the image hash codes matched by each node in the text global graph and its adjacent nodes; the reconstruction loss includes the reconstructed features The loss between the original image feature V and the reconstructed feature The loss between the original text feature T;

[0013] (S5) If the loss function has not converged, proceed to (S1) to perform training for the next training batch; otherwise, use the current first hash model and the second hash model as the image hash code extraction model and the text hash code extraction model, respectively.

[0014] Furthermore, the expression of reconstruction loss is as follows:

[0015]

[0016] Among them, L r represents the reconstruction loss; M represents the number of training data in the current training batch; Vi and They represent the original features and reconstructed features of the image in the i-th training data in the current training batch respectively; T i and Respectively represent the original features and reconstructed features of the text in the i-th training data in the current training batch; |||| 2 represents the two-norm.

[0017] Furthermore, the expression of inter-modality contrast loss is as follows:

[0018]

[0019] Among them, L inter represents the inter-modality contrast loss; N represents the number of training data in the training set; represents the adjacency matrix of the global graph of the image, represents the weight of the edge between nodes i and j in the global graph of the image, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the adjacency matrix of the global graph of text, represents the weight of the edge between nodes i and j in the global text graph, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; Represents the similarity between the i-th image hash code and the j-th text hash code in the training set.

[0020] Furthermore, the expression of intra-modal contrast loss is as follows:

[0021]

[0022] Among them, L intra represents the intra-modal contrast loss; represents the similarity between the hash code of the jth image and the hash code of the jth image in the training set; Represents the similarity between the i-th text hash code and the j-th text hash code in the training set.

[0023] Furthermore, the calculation formula for the similarity between hash codes is:

[0024]

[0025] Among them, h i and h j are two hash codes; S(h i ,h j ) is the hash code h i and h jThe similarity between i ,h j ) represents the hash code h i and h j The distance between them; γ represents the size parameter of the Cauchy distribution.

[0026] According to another aspect of the present invention, there is provided an image-text cross-modal hash code extraction method, comprising:

[0027] For the object whose hash code is to be extracted, its modality is determined. If it is an image, its continuous hash code is extracted using the image hash code extraction model and discretized; if it is a text, its continuous hash code is extracted using the text hash code extraction model and discretized;

[0028] Among them, the image hash code extraction model and the text hash code extraction model are established by the above-mentioned image-text cross-modal hash code extraction model establishment method provided by the present invention.

[0029] Furthermore, the calculation formula for discretizing the continuous hash code is:

[0030] b v =sgn(h v )

[0031] b t =sgn(h t )

[0032] Among them, h v represents the continuous hash code of the image, b v The discrete hash code representing the image; h t Represents the continuous hash code of the text, b t represents the discrete hash code of the text; sgn(·) represents the symbolic function.

[0033] According to another aspect of the present invention, a method for establishing an image-text cross-modal hash code library is provided, comprising:

[0034] For a data set consisting of mutually matching image-text pairs, the image-text cross-modal hash code extraction method provided by the present invention is used to extract discrete hash codes of each image to obtain an image hash code library, and the image-text cross-modal hash code extraction method provided by the present invention is used to extract discrete hash codes of each text to obtain a text hash code library.

[0035] According to another aspect of the present invention, there is provided an image-text cross-modal retrieval method, comprising:

[0036] (T1) For the target object to be retrieved, determine its modality. If it is an image, proceed to (T2); if it is a text, proceed to (T3);

[0037] (T2) After extracting the discrete hash code of the target object using the above-mentioned image-text cross-modal hash code extraction method provided by the present invention, the similarity between the discrete hash code and each text hash code in the text hash code library is calculated respectively, and the text corresponding to the topK text hash codes with the highest similarity is taken as the retrieval result, and transferred to (T4);

[0038] (T3) After extracting the discrete hash code of the target object using the above-mentioned image-text cross-modal hash code extraction method provided by the present invention, the similarity between the discrete hash code and each image hash code in the image hash code library is calculated respectively, and the images corresponding to the topK image hash codes with the highest similarity are taken as the retrieval results, and transferred to (T4);

[0039] (T4) Search ends;

[0040] Among them, the image hash code library and the text hash code library are established by the above-mentioned image-text cross-modal hash code library establishment method provided by the present invention; topK is a preset positive integer.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program; when the computer program is executed by a processor, the above-mentioned image-text cross-modal hash code extraction model establishment method provided by the present invention, or the above-mentioned image-text cross-modal hash code extraction method provided by the present invention, or the above-mentioned image-text cross-modal hash code library establishment method provided by the present invention, or the above-mentioned image-text cross-modal retrieval method provided by the present invention is implemented.

[0042] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0043] (1) When training the image hash code extraction model and the text hash code extraction model, the present invention uses the hash codes of the entire training data set and each training batch to construct a global graph and a local graph respectively, constructs the intra-modality contrast loss and the inter-modality contrast loss based on the global graph, and uses the local graph to reconstruct the high-dimensional features to construct the reconstruction loss, so that the training process of the image hash code extraction model and the text hash code extraction model combines the two complementary tasks of global graph contrast and local graph reconstruction, fully mines and integrates the structural information of the image and text (including global information and local information), and effectively aligns the two different modalities; in addition, in the local graph reconstruction process, the image local graph is used to reconstruct the text features, and the text local graph is used to reconstruct the image features, so as to realize cross-modal reconstruction, which can further improve the alignment effect of the two modalities and enhance the robustness of the model. Finally, the hash code extraction model established by the present invention can enable hash codes with similar semantics from different modalities to be closely mapped together, effectively reducing the modality gap, and enhancing the discriminability of the hash code, which is conducive to improving the accuracy of subsequent cross-modal retrieval.

[0044] (2) The present invention integrates global graph inter-modality contrast learning, global graph intra-modality contrast learning and local graph cross-modal reconstruction into a unified learning framework, achieving efficient integration and utilization of image and text information, and can significantly improve the retrieval performance of unsupervised cross-modal hashing methods.

[0045] (3) During the model training and cross-modal retrieval process, the present invention does not need to rely on manually annotated data and can be trained on large-scale unlabeled image and text data pairs, which solves the problem of difficulty in obtaining sufficient labels for large-scale data in practical applications and reduces training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A block diagram of a method for establishing an image-text cross-modal hash code extraction model provided by an embodiment of the present invention;

[0047] Figure 2 A flow chart of a method for establishing an image-text cross-modal hash code extraction model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0050] In order to improve the accuracy of image-text cross-modal retrieval, the present invention provides an image-text cross-modal hash code extraction model establishment method and its application. The overall concept is: in the cross-modal hash code extraction model training process, a global map and a local map are established for the training set and the training batch respectively, and on the basis of the global map and the local map, two complementary tasks of global map comparison and local map reconstruction are combined to fully mine and integrate the global-local structural information of images and texts, and effectively align the two different modalities, thereby improving the quality of the hash code extracted by the model and improving the accuracy of image-text cross-modal retrieval.

[0051] The present invention will pre-establish two hash codes, namely the first hash code and the second hash code. The first hash model is used to extract the image feature V and convert it into a continuous hash code; the second hash model is used to extract the text encoder T of the text feature and convert it into a continuous hash code. Based on the above concept, the present invention can divide the training process of the two models into three stages in each training batch, namely the global map and local map establishment stage, the global map comparison stage and the local map reconstruction stage. The specific stages are as follows:

[0052] The global and local graph building stages include:

[0053] The first hash model is used to extract continuous hash codes of images in the training set and training batches, and the second hash model is used to extract continuous hash codes of texts in the training set and training batches; for the hash codes in each batch in the training and the hash codes of the entire training set, the hash codes are used as nodes, and the similarities between the hash codes are used as the weights of the edges between the corresponding nodes. The K nearest neighbor algorithm (KNN) is used to establish local and global graphs of images and texts; for the constructed image global graph, image local graph, text global graph and text local graph, adjacency matrices can be constructed respectively.

[0054] The global map comparison phase includes:

[0055] Inter-modal alignment: Calculate the similarity of the hash codes between the image and text modalities in the training set to obtain the inter-modal similarity matrix; input the adjacency matrix of the image global graph, the adjacency matrix of the text global graph and the inter-modal similarity matrix into the contrastive learning framework to obtain the inter-modal contrast loss.

[0056] Intra-modal differentiation: Calculate the similarity between images and texts in the current training batch respectively to obtain the image similarity matrix and the text similarity matrix; input the image similarity matrix and the image global graph into the contrastive learning framework to obtain the contrastive loss within the image modality; input the text similarity matrix and the text global graph into the contrastive learning framework to obtain the contrastive loss within the text modality.

[0057] The local image reconstruction stage includes:

[0058] The image decoder is used to reconstruct the local image map into high-dimensional original features in the text space, and the text decoder is used to reconstruct the local text map into high-dimensional original features in the image space to achieve cross-modal reconstruction. The difference between the reconstructed features and the original features is calculated to obtain the reconstruction loss.

[0059] The present invention combines the above-mentioned inter-modality contrast loss, intra-modality contrast loss and reconstruction loss together to obtain the overall loss function in the model training process, and performs model training based on the loss function until the loss function converges. During the model training process, through the constraint of inter-modality contrast loss, hash codes with similar semantics from different modalities can be closely mapped together in the shared Hamming space; through the constraint of intra-modality contrast loss, hash codes with similar semantics from the same modality can be closely mapped together in the shared Hamming space; during the model training process, through the constraint of reconstruction loss, the image hash code extracted by the model can be used to accurately reconstruct the text features matching it, and the text hash code extracted by the model can be used to accurately reconstruct the image features matching it, further improving the alignment quality between different modalities. In addition, since the relevant loss utilizes both the global graph and the local graph, the model can make full use of the global structural information and the local structural information during the training process, further improving the training effect of the model. Finally, the hash code extracted by the model can effectively improve the accuracy of image-text cross-modal retrieval.

[0060] The following are examples.

[0061] Embodiment 1:

[0062] A method for establishing an image-text cross-modal hash code extraction model, such as Figure 1 and Figure 2 As shown, it includes steps (S1) to (S5). Each step is described in detail below.

[0063] (S1) A first hash model is used to extract a hash code of each image in a training set and a current training batch, and a second hash model is used to extract a hash code of each text in the training set and the current training batch; the training data in the training set and the current training batch include matching images and texts; the first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract a text encoder T of text features and convert them into continuous hash codes.

[0064] like Figure 1 As shown, in this embodiment, the first hash model specifically includes an image encoder for extracting image features V and a first hash layer for converting the image features V into continuous hash codes; the second hash model includes a text encoder for extracting text features T and a second hash model for converting the text features T into continuous hash codes;

[0065] Optionally, in this embodiment, the image encoder is a pre-trained VGGNet, and the extracted image feature V is z 1 dimensional space feature vector, the total number of features is N; the text encoder is a pre-trained bag-of-words model (BoW), and the extracted text features are z 2 dimensional space feature vector, the total number of features is N; the first hash layer and the second hash layer are both fully connected networks, and the continuous hash code obtained by converting the image feature V through the first hash layer is recorded as h v , the continuous hash code obtained by converting the text feature T through the second hash layer is recorded as h i .

[0066] It should be noted that this is only an optional implementation of the present invention and should not be understood as the sole limitation of the present invention. In other embodiments of the present invention, other image and text feature extraction models may also be used, and the hash layer for converting features into continuous hash codes may also be implemented using other network structures that can achieve the same functions.

[0067] (S2) establishing a graph structure according to the image hash code in the training set, the image hash code in the current training batch, the text hash code in the training set, and the text hash code in the current training batch, respectively, to obtain an image global graph, an image local graph, a text global graph, and a text local graph; the graph structure uses hash codes as nodes, and there are edges between each node and the K nodes with the highest similarity to it, and the weight of the edge is the similarity between the nodes; the similarity between the nodes is the similarity between the corresponding hash codes; K is a positive integer.

[0068] In this embodiment, the image global graph and the text global graph are respectively denoted as G GI and G GT For the global graph, its adjacency matrix A can be constructed, and its expression is as follows:

[0069]

[0070] Among them, A ij Indicates the adjacency between the i-th node and the j-th node, Indicates the closest to the hash code h The nearest neighbor set. At the same time, the degree matrix can be used D = diag (d 1 ,d 2 ,…,d M )express, The adjacency matrix is ​​further normalized by the following formula:

[0071]

[0072] Where I is the identity matrix, M is the number of training data in the training set; the standardized adjacency matrix satisfies: when h i and h j When we are neighbors

[0073] Through the above steps, we can obtain the normalized adjacency matrices of the image global graph and the text global graph, which are recorded as and

[0074] The image local map and text local map are denoted as G LI and G LT , then the normalized adjacency matrices of the image local graph and the text local graph can be calculated in the same way, denoted as and

[0075] (S3) Reconstructing the local image into features using the first reconstruction model Use the second reconstruction model to reconstruct the text local map into features

[0076] Optionally, in this embodiment, a graph convolutional neural network (GCN) is used as a reconstruction model to transform the image local graph G LI After inputting the reconstruction model, the features are reconstructed The text local graph G LT The normalized adjacency matrix After inputting the reconstruction model, the features are reconstructed Taking local image decoding as an example, the information transmission during the decoding process is expressed as follows:

[0077]

[0078] Among them, σ represents the activation function, and Both represent the weights of the graph convolutional neural network, and V represents the original features of the image.

[0079] (S4) establishing a loss function, and training the first hash model, the second hash model, the first reconstruction model, and the second reconstruction model according to the loss function; the loss function includes intra-modality contrast loss, inter-modality contrast loss, and reconstruction loss; the intra-modality contrast loss is used to shorten the distance between adjacent hash codes in the image global graph and the text global graph, and to lengthen the distance between non-adjacent hash codes in the image global graph and the text global graph; the inter-modality contrast loss includes the similarity loss between the text hash codes matched by each node in the image global graph and its adjacent nodes, and the similarity loss between the image hash codes matched by each node in the text global graph and its adjacent nodes; the reconstruction loss includes the reconstructed features The loss between the original image feature V and the reconstructed feature The loss between the feature T and the original text.

[0080] In this embodiment, the expression of reconstruction loss is as follows:

[0081]

[0082] Among them, L r represents the reconstruction loss; M represents the number of training data in the current training batch; V i and They represent the original features and reconstructed features of the image in the i-th training data in the current training batch respectively; T i and Respectively represent the original features and reconstructed features of the text in the i-th training data in the current training batch; |||| 2 represents the two-norm;

[0083] The expression of inter-modality contrast loss is as follows:

[0084]

[0085]

[0086] Among them, L inter represents the inter-modality contrast loss; N represents the number of training data in the training set; represents the adjacency matrix of the global graph of the image, represents the weight of the edge between nodes i and j in the global graph of the image, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the adjacency matrix of the global graph of text, represents the weight of the edge between nodes i and j in the global text graph, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the similarity between the i-th image hash code and the j-th text hash code in the training set;

[0087] The expression of intra-modal contrast loss is as follows:

[0088]

[0089] Among them, L intra represents the intra-modal contrast loss; represents the similarity between the hash code of the jth image and the hash code of the jth image in the training set; Represents the similarity between the i-th text hash code and the j-th text hash code in the training set.

[0090] (S5) If the loss function has not converged, proceed to (S1) to perform training for the next training batch; otherwise, use the current first hash model and the second hash model as the image hash code extraction model and the text hash code extraction model, respectively.

[0091] Finally, the overall loss function can be expressed as follows:

[0092] L=λL intra +L inter +L r

[0093] Among them, λ is a trade-off parameter.

[0094] Optionally, this embodiment calculates the similarity between hash codes by the following formula:

[0095]

[0096] Among them, h i and h j are two hash codes; S(h i ,h j ) is the hash code h i and h j The similarity between i ,h j ) represents the hash code h i and h j The distance between them; γ represents the size parameter of the Cauchy distribution.

[0097] In general, this embodiment first uses the global structural information of all training set data to establish a global graph of each modality, and uses the local structural information of each batch of training data to establish a local graph of each modality, making full use of the graph structure information of the data to enhance the robustness of the model and the discriminability of hash codes; on the basis of the global graph and the local graph, by combining the global graph inter-modal comparison learning task and the local graph cross-modal reconstruction task, the model is helped to learn a shared Hamming space, in which hash codes from different modalities with similar semantics can be closely mapped together, effectively reducing the modality gap. The hash codes extracted by the hash code extraction model finally trained can effectively improve the accuracy of image-text cross-modal retrieval.

[0098] Embodiment 2:

[0099] An image-text cross-modal hash code extraction method, comprising:

[0100] For the object whose hash code is to be extracted, its modality is determined. If it is an image, its continuous hash code is extracted using the image hash code extraction model and discretized; if it is a text, its continuous hash code is extracted using the text hash code extraction model and discretized;

[0101] The image hash code extraction model and the text hash code extraction model are established by the image-text cross-modal hash code extraction model establishment method provided in the above-mentioned embodiment 1.

[0102] Optionally, in this embodiment, the calculation formula for discretizing continuous hash codes is:

[0103] b v =sgn(h v )

[0104] b t =sgn(h t )

[0105] Among them, h v represents the continuous hash code of the image, b v The discrete hash code representing the image; h t Represents the continuous hash code of the text, b t represents the discrete hash code of the text; sgn(·) represents the symbolic function.

[0106] Embodiment 3:

[0107] A method for establishing an image-text cross-modal hash code library, comprising:

[0108] For a data set consisting of mutually matching image-text pairs, the image-text cross-modal hash code extraction method provided in the above embodiment 2 is used to extract the discrete hash code of each image to obtain an image hash code library, and the image-text cross-modal hash code extraction method provided in the above embodiment 2 is used to extract the discrete hash code of each text to obtain a text hash code library.

[0109] Embodiment 4:

[0110] An image-text cross-modal retrieval method, comprising:

[0111] (T1) For the target object to be retrieved, determine its modality. If it is an image, proceed to (T2); if it is a text, proceed to (T3);

[0112] (T2) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method provided in the above embodiment 2, the similarity between the discrete hash code and each text hash code in the text hash code library is calculated respectively, and the text corresponding to the topK text hash codes with the highest similarity is taken as the retrieval result, and transferred to (T4);

[0113] (T3) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method provided in the above-mentioned embodiment 2, the similarity between the discrete hash code and each image hash code in the image hash code library is calculated respectively, and the images corresponding to the topK image hash codes with the highest similarity are taken as the retrieval results, and transferred to (T4);

[0114] (T4) Search ends;

[0115] The image hash code library and the text hash code library are established by the image-text cross-modal hash code library establishment method provided in the above-mentioned embodiment 3; topK is a preset positive integer.

[0116] Embodiment 5:

[0117] A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, the image-text cross-modal hash code extraction model establishment method provided in the above embodiment 1, or the image-text cross-modal hash code extraction method provided in the above embodiment 2, or the image-text cross-modal hash code library establishment method provided in the above embodiment 3, or the image-text cross-modal retrieval method provided in the above embodiment 4 is implemented.

[0118] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for establishing an image-text cross-modal hash code extraction model, characterized in that: include: (S1) using a first hash model to extract hash codes of images in a training set and a current training batch, and using a second hash model to extract hash codes of texts in the training set and the current training batch; the training data in the training set and the current training batch include matching images and texts; the first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract a text encoder T of text features and convert them into continuous hash codes; (S2) establishing a graph structure according to the image hash code in the training set, the image hash code in the current training batch, the text hash code in the training set, and the text hash code in the current training batch, respectively, to obtain an image global graph, an image local graph, a text global graph, and a text local graph; the graph structure uses hash codes as nodes, and there are edges between each node and the K nodes with the highest similarity to it, and the weight of the edge is the similarity between the nodes; the similarity between the nodes is the similarity between the corresponding hash codes; K is a positive integer; (S3) Reconstructing the local image into features using the first reconstruction model Use the second reconstruction model to reconstruct the text local map into features (S4) establishing a loss function, and training the first hash model, the second hash model, the first reconstruction model and the second reconstruction model according to the loss function; the loss function includes intra-modality contrast loss, inter-modality contrast loss and reconstruction loss; the intra-modality contrast loss is used to shorten the distance between adjacent hash codes in the image global graph and the text global graph, and to lengthen the distance between non-adjacent hash codes in the image global graph and the text global graph; the inter-modality contrast loss includes the similarity loss between the text hash codes matched by each node in the image global graph and its adjacent nodes, and the similarity loss between the image hash codes matched by each node in the text global graph and its adjacent nodes; the reconstruction loss includes the reconstructed features The loss between the original image feature V and the reconstructed feature The loss between the original text feature T; (S5) If the loss function has not converged, proceed to (S1) to perform training for the next training batch; otherwise, use the current first hash model and the second hash model as the image hash code extraction model and the text hash code extraction model, respectively.

2. The method for establishing an image-text cross-modal hash code extraction model as described in claim 1, characterized in that: The expression of the reconstruction loss is as follows: Among them, L r represents the reconstruction loss; M represents the number of training data in the current training batch; V i and They represent the original features and reconstructed features of the image in the i-th training data in the current training batch respectively; T i and They respectively represent the original features and the reconstructed features of the text in the i-th training data in the current training batch; ||||2 represents the binary norm.

3. The method for establishing an image-text cross-modal hash code extraction model as claimed in claim 1, characterized in that: The expression of the inter-modality contrast loss is as follows: Among them, L inter represents the inter-modality contrast loss; N represents the number of training data in the training set; represents the adjacency matrix of the global graph of the image, represents the weight of the edge between nodes i and j in the global graph of the image, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the adjacency matrix of the global graph of text, represents the weight of the edge between nodes i and j in the global text graph, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; Represents the similarity between the i-th image hash code and the j-th text hash code in the training set.

4. The method for establishing an image-text cross-modal hash code extraction model as described in claim 3, characterized in that: The expression of the intra-modal contrast loss is as follows: Among them, L intra represents the intra-modal contrast loss; represents the similarity between the hash code of the jth image and the hash code of the ,th image in the training set; Represents the similarity between the i-th text hash code and the ,th text hash code in the training set.

5. The method for establishing an image-text cross-modal hash code extraction model according to any one of claims 1 to 4, characterized in that: The calculation formula for the similarity between hash codes is: Among them, h i and h j are two hash codes; S(h i ,h j ) is the hash code h i and h j The similarity between i ,h j ) represents the hash code h i and h j The distance between them; γ represents the size parameter of the Cauchy distribution.

6. A method for extracting cross-modal hash codes from image to text, characterized in that: include: For the object whose hash code is to be extracted, its modality is determined. If it is an image, its continuous hash code is extracted using the image hash code extraction model and discretized; if it is a text, its continuous hash code is extracted using the text hash code extraction model and discretized; The image hash code extraction model and the text hash code extraction model are established by the image-text cross-modal hash code extraction model establishment method according to any one of claims 1 to 5.

7. The image-text cross-modal hash code extraction method according to claim 6, characterized in that: The calculation formula for discretizing continuous hash codes is: b v =sgn(h v ) b t =sgn(h t ) Among them, h v represents the continuous hash code of the image, b v The discrete hash code representing the image; h t Represents the continuous hash code of the text, h t represents the discrete hash code of the text; sgn(·) represents the symbolic function.

8. A method for establishing an image-text cross-modal hash code library, characterized in that: include: For a data set consisting of mutually matching image-text pairs, the image-text cross-modal hash code extraction method described in claim 6 or 7 is used to extract the discrete hash code of each image to obtain an image hash code library, and the image-text cross-modal hash code extraction method described in claim 6 or 7 is used to extract the discrete hash code of each text to obtain a text hash code library.

9. An image-text cross-modal retrieval method, characterized in that: include: (T1) For the target object to be retrieved, determine its modality. If it is an image, proceed to (T2); if it is a text, proceed to (T3); (T2) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method described in claim 6 or 7, respectively calculate the similarity between the discrete hash code and each text hash code in the text hash code library, and take the text corresponding to the topK text hash codes with the highest similarity as the retrieval result, and proceed to (T4); (T3) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method described in claim 6 or 7, respectively calculate the similarity between the discrete hash code and each image hash code in the image hash code library, and take the images corresponding to the topK image hash codes with the highest similarity as the retrieval results, and proceed to (T4); (T4) Search ends; Wherein, the image hash code library and the text hash code library are established by the image-text cross-modal hash code library establishment method described in claim 8; topK is a preset positive integer.

10. A computer-readable storage medium, characterized in that: Comprising a stored computer program; when the computer program is executed by a processor, it implements the method for establishing an image-text cross-modal hash code extraction model as described in any one of claims 1 to 5, or the image-text cross-modal hash code extraction method as described in claim 6 or 7, or the image-text cross-modal hash code library establishment method as described in claim 8, or the image-text cross-modal retrieval method as described in claim 9.

Images