Image-text cross-modal hash code extraction model establishment method and its application

Abstract

The present invention discloses a method for establishing an image-text cross-modal hash code extraction model and its application, belonging to the field of cross-modal retrieval. The method comprises: using a hash model to extract hash codes for images and texts in a training set and a current training batch, respectively; establishing a global image graph, a local image graph, a global text graph, and a local text graph; reconstructing text features using the local image graph, and reconstructing image features using the global text graph; establishing a loss function, and training the model until the loss function converges; the loss function comprises: an intra-modal contrast loss, which is used to shorten the distance between adjacent hash codes in the global graph and increase the distance between non-adjacent hash codes; an inter-modal contrast loss, which comprises the similarity loss between each node in the global graph and the hash codes of the other modality matched by its adjacent nodes; and a reconstruction loss, which comprises the loss between the reconstructed features and the original features. The present invention can improve the accuracy of image-text cross-modal retrieval.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal retrieval, and more specifically, relates to a method for establishing an image-text cross-modal hash code extraction model and its application. Background Art

[0002] Efficient cross-modal retrieval of image and text data, which involves retrieving similar text from an image or similar images from a text, has important applications in multimedia retrieval and processing. Cross-modal hashing methods, which can compress high-dimensional features into binary codes, reduce memory usage while improving retrieval efficiency, have become a mainstream solution in cross-modal retrieval. Unsupervised cross-modal hashing methods, which do not require annotation of large datasets, have become a mainstream solution in this area.

[0003] However, existing cross-image-text modality unsupervised hashing methods still have the following limitations:

[0004] (1) Insufficient modality alignment: Existing methods aim to improve the accuracy of the supervision signal but cannot effectively bridge the gap between modalities, resulting in suboptimal performance;

[0005] (2) Incomplete utilization of structural information: Existing methods only use implicit graph structures to establish more accurate pairwise similarities, ignoring the global and local information of the graph, which in turn affects the accuracy of cross-modal retrieval.

[0006] Overall, the existing hashing methods for image-text cross-modal retrieval need to be further improved in terms of the accuracy of cross-modal retrieval. Summary of the Invention

[0007] In response to the defects and improvement needs of the existing technology, the present invention provides a method for establishing an image-text cross-modal hash code extraction model and its application. Its purpose is to fully utilize the global and local information of the graph and achieve modal alignment during the training process of the hash code extraction model, so that the hash codes extracted by the hash code extraction model, hash codes from different modalities with similar semantics can be closely mapped together, and the discriminability of the hash codes is enhanced, thereby improving the accuracy of image-text cross-modal retrieval.

[0008] To achieve the above objectives, according to one aspect of the present invention, a method for establishing an image-text cross-modal hash code extraction model is provided, comprising:

[0009] (S1) using a first hash model to extract hash codes for each image in a training set and a current training batch, and using a second hash model to extract hash codes for each text in the training set and the current training batch; the training data in the training set and the current training batch include matching images and texts; the first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract a text encoder T for text features and convert them into continuous hash codes;

[0010] (S2) establishing a graph structure based on the image hash codes in the training set, the image hash codes in the current training batch, the text hash codes in the training set, and the text hash codes in the current training batch, respectively, to obtain an image global graph, an image local graph, a text global graph, and a text local graph; the graph structure uses hash codes as nodes, and there are edges between each node and the K nodes with the highest similarity to it, and the weight of the edge is the similarity between the nodes; the similarity between the nodes is the similarity between the corresponding hash codes; K is a positive integer;

[0011] (S3) Reconstructing the local image into features using the first reconstruction model Use the second reconstruction model to reconstruct the text local map into features

[0012] (S4) establishing a loss function, and training the first hash model, the second hash model, the first reconstruction model, and the second reconstruction model according to the loss function; the loss function includes intra-modality contrast loss, inter-modality contrast loss, and reconstruction loss; the intra-modality contrast loss is used to shorten the distance between adjacent hash codes in the image global graph and the text global graph, and to lengthen the distance between non-adjacent hash codes in the image global graph and the text global graph; the inter-modality contrast loss includes the similarity loss between the text hash codes matched by each node in the image global graph and its adjacent nodes, and the similarity loss between the image hash codes matched by each node in the text global graph and its adjacent nodes; the reconstruction loss includes the reconstructed features The loss between the original image feature V and the reconstructed feature The loss between the original text feature T;

[0013] (S5) If the loss function has not converged, proceed to (S1) to perform training for the next training batch; otherwise, use the current first hash model and the second hash model as the image hash code extraction model and the text hash code extraction model, respectively.

[0014] Furthermore, the reconstruction loss is expressed as follows:

[0015]

[0016] Among them, L r Represents the reconstruction loss; M represents the number of training data in the current training batch; Vi and Respectively represent the original features and reconstructed features of the image in the i-th training data in the current training batch; T i and They represent the original features and reconstructed features of the text in the i-th training data in the current training batch respectively; ||||2 represents the two-norm.

[0017] Furthermore, the expression of inter-modality contrast loss is as follows:

[0018]

[0019] Among them, L inter represents the inter-modality contrast loss; N represents the number of training data in the training set; represents the adjacency matrix of the global graph of the image, represents the weight of the edge between nodes i and j in the global graph of the image, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the adjacency matrix of the global graph of text, represents the weight of the edge between nodes i and j in the global text graph, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; Represents the similarity between the hash code of the i-th image and the hash code of the j-th text in the training set.

[0020] Furthermore, the expression of intra-modal contrast loss is as follows:

[0021]

[0022] Among them, L intra represents the intra-modality contrast loss; represents the similarity between the hash code of the jth image and the hash code of the jth image in the training set; Represents the similarity between the hash code of the i-th text and the hash code of the j-th text in the training set.

[0023] Furthermore, the calculation formula for the similarity between hash codes is:

[0024]

[0025] Among them, h i and h j are two hash codes; S(h i ,h j ) is the hash code h i and h j The similarity between d(hi ,h j ) represents the hash code h i and h j The distance between them; γ represents the size parameter of the Cauchy distribution.

[0026] According to another aspect of the present invention, a method for extracting cross-modal hash codes from an image and text is provided, comprising:

[0027] For the object whose hash code is to be extracted, its modality is determined. If it is an image, its continuous hash code is extracted using the image hash code extraction model and discretized. If it is text, its continuous hash code is extracted using the text hash code extraction model and discretized.

[0028] Among them, the image hash code extraction model and the text hash code extraction model are established by the above-mentioned image-text cross-modal hash code extraction model establishment method provided by the present invention.

[0029] Furthermore, the calculation formula for discretizing the continuous hash code is:

[0030] b v =sgn(h v )

[0031] b t =sgn(h t )

[0032] Among them, h v Represents the continuous hash code of the image, b v The discrete hash code representing the image; h t Represents the continuous hash code of the text, b t represents the discrete hash code of the text; sgn(·) represents the symbolic function.

[0033] According to another aspect of the present invention, a method for establishing an image-text cross-modal hash code library is provided, comprising:

[0034] For a data set consisting of mutually matching image-text pairs, the above-mentioned image-text cross-modal hash code extraction method provided by the present invention is used to extract the discrete hash code of each image to obtain an image hash code library, and the above-mentioned image-text cross-modal hash code extraction method provided by the present invention is used to extract the discrete hash code of each text to obtain a text hash code library.

[0035] According to another aspect of the present invention, there is provided an image-text cross-modal retrieval method, comprising:

[0036] (T1) For the target object to be retrieved, determine its modality. If it is an image, proceed to (T2); if it is a text, proceed to (T3);

[0037] (T2) After extracting a discrete hash code of the target object using the image-text cross-modal hash code extraction method provided by the present invention, similarities between the discrete hash code and each text hash code in the text hash code library are calculated, and the texts corresponding to the topK text hash codes with the highest similarity are taken as the retrieval results, and transferred to (T4);

[0038] (T3) After extracting the discrete hash code of the target object using the above-mentioned image-text cross-modal hash code extraction method provided by the present invention, the similarity between the discrete hash code and the hash code of each image in the image hash code library is calculated respectively, and the images corresponding to the topK image hash codes with the highest similarity are taken as the retrieval results and transferred to (T4);

[0039] (T4) Search ends;

[0040] Among them, the image hash code library and the text hash code library are established by the above-mentioned image-text cross-modal hash code library establishment method provided by the present invention; topK is a preset positive integer.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program; when the computer program is executed by a processor, the method for establishing the image-text cross-modal hash code extraction model provided by the present invention, or the method for extracting the image-text cross-modal hash code provided by the present invention, or the method for establishing the image-text cross-modal hash code library provided by the present invention, or the method for retrieving the image-text cross-modal hash code provided by the present invention is implemented.

[0042] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0043] (1) When training the image hash code extraction model and the text hash code extraction model, the present invention uses the hash codes of the entire training data set and each training batch to construct a global graph and a local graph respectively, constructs the intra-modal contrast loss and the inter-modal contrast loss based on the global graph, and uses the local graph to reconstruct high-dimensional features to construct the reconstruction loss, so that the training process of the image hash code extraction model and the text hash code extraction model combines the two complementary tasks of global graph comparison and local graph reconstruction, fully mines and integrates the structural information of the image and text (including global information and local information), and effectively aligns the two different modalities; in addition, in the local graph reconstruction process, the image local graph is used to reconstruct the text features, and the text local graph is used to reconstruct the image features, realizing cross-modal reconstruction, which can further improve the alignment effect of the two modalities and enhance the robustness of the model. Ultimately, the hash code extraction model established by the present invention can enable hash codes with similar semantics from different modalities to be closely mapped together, effectively reducing the modality gap and enhancing the discriminability of the hash code, which is conducive to improving the accuracy of subsequent cross-modal retrieval.

[0044] (2) The present invention integrates global graph inter-modality contrast learning, global graph intra-modality contrast learning and local graph cross-modal reconstruction into a unified learning framework, achieving efficient integration and utilization of image and text information, and can significantly improve the retrieval performance of unsupervised cross-modal hashing methods.

[0045] (3) During the model training and cross-modal retrieval process, the present invention does not need to rely on manually labeled data and can be trained on large-scale unlabeled image and text data pairs, which solves the problem of difficulty in obtaining sufficient labels for large-scale data in practical applications and reduces training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A block diagram of a method for establishing an image-text cross-modal hash code extraction model provided by an embodiment of the present invention;

[0047] Figure 2 Flowchart of a method for establishing an image-text cross-modal hash code extraction model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0049] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0050] In order to improve the accuracy of image-text cross-modal retrieval, the present invention provides a method for establishing an image-text cross-modal hash code extraction model and its application. The overall concept is: during the training process of the cross-modal hash code extraction model, a global graph and a local graph are established for the training set and training batch respectively, and on the basis of the global graph and the local graph, two complementary tasks, global graph comparison and local graph reconstruction, are combined to fully mine and integrate the global-local structural information of images and texts, while effectively aligning the two different modalities, thereby improving the quality of the hash code extracted by the model and improving the accuracy of image-text cross-modal retrieval.

[0051] The present invention pre-establishes two hash codes, namely the first hash code and the second hash code. The first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract text features from the text encoder T and convert them into continuous hash codes. Based on the above concept, the present invention divides the training process of the two models in each training batch into three stages, namely the global and local map establishment stage, the global map comparison stage, and the local map reconstruction stage. The details of each stage are as follows:

[0052] The global and local graph creation phases include:

[0053] A first hash model is used to extract continuous hash codes of images in a training set and a training batch, and a second hash model is used to extract continuous hash codes of texts in a training set and a training batch. For the hash codes in each batch during training and the hash codes of the entire training set, the hash codes are used as nodes, and the similarity between the hash codes is used as the weight of the edges between the corresponding nodes. The K-nearest neighbor algorithm (KNN) is used to establish local and global graphs of images and texts. For the constructed global image graph, local image graph, global text graph, and local text graph, adjacency matrices can be constructed respectively.

[0054] The global map comparison phase includes:

[0055] Inter-modal alignment: Calculate the similarity between the hash codes of the image and text modalities in the training set to obtain the inter-modal similarity matrix; input the adjacency matrix of the image global graph, the adjacency matrix of the text global graph, and the inter-modal similarity matrix into the contrastive learning framework to obtain the inter-modal contrastive loss.

[0056] Intra-modality discrimination: Calculate the similarity between images and texts in the current training batch to obtain the image similarity matrix and the text similarity matrix respectively. Input the image similarity matrix and the image global graph into the contrastive learning framework to obtain the contrastive loss within the image modality. Input the text similarity matrix and the text global graph into the contrastive learning framework to obtain the contrastive loss within the text modality.

[0057] The local image reconstruction stage includes:

[0058] The image decoder is used to reconstruct the local image map into high-dimensional original features in the text space, and the text decoder is used to reconstruct the local text map into high-dimensional original features in the image space to achieve cross-modal reconstruction. The difference between the reconstructed features and the original features is calculated to obtain the reconstruction loss.

[0059] The present invention combines the above-mentioned inter-modality contrast loss, intra-modality contrast loss, and reconstruction loss to obtain an overall loss function during model training. Model training is then performed based on this loss function until the loss function converges. During model training, the constraints of the inter-modality contrast loss allow hash codes from different modalities with similar semantics to be tightly mapped together in a shared Hamming space. The constraints of the intra-modality contrast loss allow hash codes from the same modality with similar semantics to be tightly mapped together in a shared Hamming space. During model training, the constraints of the reconstruction loss allow the image hash codes extracted by the model to be used to accurately reconstruct the matching text features, and the text hash codes extracted by the model to be used to accurately reconstruct the matching image features, further improving the alignment quality between different modalities. Furthermore, because the associated losses utilize both the global and local graphs, the model can fully utilize both global and local structural information during training, further improving the model's training effectiveness. Ultimately, the hash codes extracted by the model can effectively improve the accuracy of image-text cross-modal retrieval.

[0060] The following are examples.

[0061] Example 1:

[0062] A method for establishing an image-text cross-modal hash code extraction model, such as Figure 1 and Figure 2 As shown, it includes steps (S1) to (S5). Each step is described in detail below.

[0063] (S1) using a first hash model to extract hash codes for images in a training set and a current training batch, and using a second hash model to extract hash codes for texts in the training set and the current training batch; the training data in the training set and the current training batch include matching images and texts; the first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract a text encoder T for text features and convert them into continuous hash codes.

[0064] like Figure 1 As shown, in this embodiment, the first hash model specifically includes an image encoder for extracting image features V and a first hash layer for converting the image features V into continuous hash codes; the second hash model includes a text encoder for extracting text features T and a second hash model for converting the text features T into continuous hash codes;

[0065] Optionally, in this embodiment, the image encoder is a pre-trained VGGNet, the extracted image feature V is a z1-dimensional space feature vector, and the total number of features is N; the text encoder is a pre-trained bag-of-words (BoW) model, the extracted text feature is a z2-dimensional space feature vector, and the total number of features is N; the first hash layer and the second hash layer are both fully connected networks, and the continuous hash code obtained by converting the image feature V through the first hash layer is recorded as h v , the continuous hash code obtained by converting the text feature T into the second hash layer is recorded as h i .

[0066] It should be noted that this is only an optional implementation of the present invention and should not be understood as the sole limitation of the present invention. In other embodiments of the present invention, other image and text feature extraction models may also be used, and the hash layer for converting features into continuous hash codes may also be implemented using other network structures that can achieve the same function.

[0067] (S2) establishing a graph structure based on the image hash codes in the training set, the image hash codes in the current training batch, the text hash codes in the training set, and the text hash codes in the current training batch, respectively, to obtain an image global graph, an image local graph, a text global graph, and a text local graph; the graph structure uses hash codes as nodes, and there are edges between each node and the K nodes with the highest similarity to it, and the weight of the edge is the similarity between the nodes; the similarity between the nodes is the similarity between the corresponding hash codes; K is a positive integer.

[0068] In this embodiment, the image global graph and the text global graph are respectively denoted as G GI and G GT For the global graph, its adjacency matrix A can be constructed, and its expression is as follows:

[0069]

[0070] Among them, A ij Indicates the adjacency between the i-th node and the j-th node, Represents the closest to the hash code h The nearest neighbor set. At the same time, the degree matrix can be used D = diag (d1, d2, ..., d M )express, The adjacency matrix is ​​further normalized by the following formula:

[0071]

[0072] Where I is the identity matrix, M is the number of training data in the training set; the standardized adjacency matrix satisfies: when h i and h j When we are neighbors

[0073] Through the above steps, we can obtain the normalized adjacency matrices of the image global graph and the text global graph, which are respectively denoted as and

[0074] The image local map and text local map are denoted as G LI and G LT , then the normalized adjacency matrices of the image local graph and the text local graph can be calculated in the same way, and are recorded as and

[0075] (S3) Reconstructing the local image into features using the first reconstruction model Use the second reconstruction model to reconstruct the text local map into features

[0076] Optionally, in this embodiment, a graph convolutional neural network (GCN) is used as a reconstruction model to transform the image local graph G LI After inputting the reconstruction model, the features are reconstructed The text local graph G LT The normalized adjacency matrix After inputting the reconstruction model, the features are reconstructed Taking local image decoding as an example, the information transmission during the decoding process is expressed as follows:

[0077]

[0078] Among them, σ represents the activation function, and Both represent the weights of the graph convolutional neural network, and V represents the original features of the image.

[0079] (S4) establishing a loss function, and training the first hash model, the second hash model, the first reconstruction model, and the second reconstruction model according to the loss function; the loss function includes intra-modality contrast loss, inter-modality contrast loss, and reconstruction loss; the intra-modality contrast loss is used to shorten the distance between adjacent hash codes in the image global graph and the text global graph, and to lengthen the distance between non-adjacent hash codes in the image global graph and the text global graph; the inter-modality contrast loss includes the similarity loss between the text hash codes matched by each node in the image global graph and its adjacent nodes, and the similarity loss between the image hash codes matched by each node in the text global graph and its adjacent nodes; the reconstruction loss includes the reconstructed features The loss between the original image feature V and the reconstructed feature The loss between the feature T and the original text.

[0080] In this embodiment, the expression of reconstruction loss is as follows:

[0081]

[0082] Among them, L r Represents the reconstruction loss; M represents the number of training data in the current training batch; V i and Respectively represent the original features and reconstructed features of the image in the i-th training data in the current training batch; T i and Respectively represent the original features and reconstructed features of the text in the i-th training data in the current training batch; ||||2 represents the two-norm;

[0083] The expression of inter-modality contrast loss is as follows:

[0084]

[0085]

[0086] Among them, L inter represents the inter-modality contrast loss; N represents the number of training data in the training set; represents the adjacency matrix of the global graph of the image, represents the weight of the edge between nodes i and j in the global graph of the image, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the adjacency matrix of the global graph of text, represents the weight of the edge between nodes i and j in the global text graph, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the similarity between the hash code of the i-th image and the hash code of the j-th text in the training set;

[0087] The expression of intra-modal contrast loss is as follows:

[0088]

[0089] Among them, L intra represents the intra-modality contrast loss; represents the similarity between the hash code of the jth image and the hash code of the jth image in the training set; Represents the similarity between the hash code of the i-th text and the hash code of the j-th text in the training set.

[0090] (S5) If the loss function has not converged, proceed to (S1) to perform training for the next training batch; otherwise, use the current first hash model and the second hash model as the image hash code extraction model and the text hash code extraction model, respectively.

[0091] Finally, the overall loss function can be expressed as follows:

[0092] L=λL intra +L inter +L r

[0093] Among them, λ is a trade-off parameter.

[0094] Optionally, this embodiment calculates the similarity between hash codes using the following formula:

[0095]

[0096] Among them, h i and h j are two hash codes; S(h i ,h j ) is the hash code h i and h j The similarity between d(h i ,h j ) represents the hash code h i and h j The distance between them; γ represents the size parameter of the Cauchy distribution.

[0097] In general, this embodiment first uses the global structural information of all training set data to establish a global graph for each modality, and then uses the local structural information of each batch of training data to establish a local graph for each modality. This fully utilizes the graph structural information of the data to enhance the robustness of the model and the discriminability of hash codes. Based on the global graph and local graphs, by combining the global graph inter-modality comparison learning task and the local graph cross-modal reconstruction task, the model learns a shared Hamming space in which hash codes from different modalities with similar semantics can be closely mapped together, effectively reducing the modality gap. The hash codes extracted by the hash code extraction model finally trained can effectively improve the accuracy of image-text cross-modal retrieval.

[0098] Example 2:

[0099] An image-text cross-modal hash code extraction method, comprising:

[0100] For the object whose hash code is to be extracted, its modality is determined. If it is an image, its continuous hash code is extracted using the image hash code extraction model and discretized. If it is text, its continuous hash code is extracted using the text hash code extraction model and discretized.

[0101] The image hash code extraction model and the text hash code extraction model are established by the image-text cross-modal hash code extraction model establishment method provided in the above-mentioned embodiment 1.

[0102] Optionally, in this embodiment, the calculation formula for discretizing the continuous hash code is:

[0103] b v =sgn(h v )

[0104] b t =sgn(h t )

[0105] Among them, h v Represents the continuous hash code of the image, b v The discrete hash code representing the image; h t Represents the continuous hash code of the text, b t represents the discrete hash code of the text; sgn(·) represents the symbolic function.

[0106] Example 3:

[0107] A method for establishing an image-text cross-modal hash code library, comprising:

[0108] For a data set consisting of matching image-text pairs, the image-text cross-modal hash code extraction method provided in the above embodiment 2 is used to extract the discrete hash code of each image to obtain an image hash code library, and the image-text cross-modal hash code extraction method provided in the above embodiment 2 is used to extract the discrete hash code of each text to obtain a text hash code library.

[0109] Example 4:

[0110] An image-text cross-modal retrieval method, comprising:

[0111] (T1) For the target object to be retrieved, determine its modality. If it is an image, proceed to (T2); if it is a text, proceed to (T3);

[0112] (T2) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method provided in the above embodiment 2, the similarity between the discrete hash code and each text hash code in the text hash code library is calculated respectively, and the texts corresponding to the topK text hash codes with the highest similarity are taken as the retrieval results, and the process proceeds to (T4);

[0113] (T3) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method provided in the above-mentioned embodiment 2, the similarity between the discrete hash code and the hash code of each image in the image hash code library is calculated respectively, and the images corresponding to the topK image hash codes with the highest similarity are taken as the retrieval results, and the process proceeds to (T4);

[0114] (T4) Search ends;

[0115] The image hash code library and the text hash code library are established by the image-text cross-modal hash code library establishment method provided in the above embodiment 3; topK is a preset positive integer.

[0116] Example 5:

[0117] A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, the computer program implements the image-text cross-modal hash code extraction model establishment method provided in the above-mentioned embodiment 1, or the image-text cross-modal hash code extraction method provided in the above-mentioned embodiment 2, or the image-text cross-modal hash code library establishment method provided in the above-mentioned embodiment 3, or the image-text cross-modal retrieval method provided in the above-mentioned embodiment 4.

[0118] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for establishing an image-text cross-modal hash code extraction model, characterized in that: include: (S1) using a first hash model to extract hash codes for each image in a training set and a current training batch, and using a second hash model to extract hash codes for each text in the training set and the current training batch; the training data in the training set and the current training batch include matching images and texts; the first hash model is used to extract image features V and convert them into continuous hash codes; the second hash model is used to extract a text encoder T for text features and convert them into continuous hash codes; (S2) establishing a graph structure based on the image hash codes in the training set, the image hash codes in the current training batch, the text hash codes in the training set, and the text hash codes in the current training batch, respectively, to obtain an image global graph, an image local graph, a text global graph, and a text local graph; the graph structure uses hash codes as nodes, and there are edges between each node and the K nodes with the highest similarity to it, and the weight of the edge is the similarity between the nodes; the similarity between the nodes is the similarity between the corresponding hash codes; K is a positive integer; (S3) Reconstructing the local image into features using the first reconstruction model Use the second reconstruction model to reconstruct the text local map into features (S4) establishing a loss function, and training the first hash model, the second hash model, the first reconstruction model and the second reconstruction model according to the loss function; the loss function includes intra-modality contrast loss, inter-modality contrast loss and reconstruction loss; the intra-modality contrast loss is used to shorten the distance between adjacent hash codes in the image global graph and the text global graph, and to lengthen the distance between non-adjacent hash codes in the image global graph and the text global graph; the inter-modality contrast loss includes the similarity loss between the text hash codes matched by each node in the image global graph and its adjacent nodes, and the similarity loss between the image hash codes matched by each node in the text global graph and its adjacent nodes; the reconstruction loss includes the reconstructed features The loss between the original image feature V and the reconstructed feature The loss between the original text feature T; (S5) If the loss function has not converged, proceed to (S1) to perform training for the next training batch; otherwise, use the current first hash model and the second hash model as the image hash code extraction model and the text hash code extraction model, respectively.

2. The method for establishing an image-text cross-modal hash code extraction model according to claim 1, wherein: The expression of the reconstruction loss is as follows: Among them, L r Represents the reconstruction loss; M represents the number of training data in the current training batch; V i and Respectively represent the original features and reconstructed features of the image in the i-th training data in the current training batch; T i and They represent the original features and reconstructed features of the text in the i-th training data in the current training batch respectively; ||||2 represents the two-norm.

3. The method for establishing an image-text cross-modal hash code extraction model according to claim 1, wherein: The expression of the inter-modality contrast loss is as follows: Among them, L inter represents the inter-modality contrast loss; N represents the number of training data in the training set; represents the adjacency matrix of the global graph of the image, represents the weight of the edge between nodes i and j in the global graph of the image, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the adjacency matrix of the global graph of text, represents the weight of the edge between nodes i and j in the global text graph, Indicates that there is an edge between nodes i and j. Indicates that there is no edge between nodes i and j; represents the similarity between the i-th image hash code and the j-th text hash code in the training set.

4. The method for establishing an image-text cross-modal hash code extraction model according to claim 3, wherein: The expression of the intra-modal contrast loss is as follows: Among them, L intra represents the intra-modality contrast loss; represents the similarity between the hash code of the jth image and the hash code of the ,th image in the training set; Represents the similarity between the i-th text hash code and the i-th text hash code in the training set.

5. The method for establishing an image-text cross-modal hash code extraction model according to any one of claims 1 to 4, wherein: The formula for calculating the similarity between hash codes is: Among them, h i and h j are two hash codes; S(h i , h j ) is the hash code h i and h j The similarity between d(h i , h j ) represents the hash code h i and h j The distance between them; γ represents the size parameter of the Cauchy distribution.

6. A method for extracting cross-modal hash codes from images and texts, characterized in that: include: For the object whose hash code is to be extracted, its modality is determined. If it is an image, its continuous hash code is extracted using the image hash code extraction model and discretized. If it is text, its continuous hash code is extracted using the text hash code extraction model and discretized. The image hash code extraction model and the text hash code extraction model are established by the image-text cross-modal hash code extraction model establishment method according to any one of claims 1 to 5.

7. The image-text cross-modal hash code extraction method according to claim 6, characterized in that: The calculation formula for discretizing continuous hash codes is: b v =sgn(h v ) b t =sgn(h t ) Among them, h v Represents the continuous hash code of the image, b v The discrete hash code representing the image; h t Represents the continuous hash code of the text, h t represents the discrete hash code of the text; sgn(·) represents the symbolic function.

8. A method for establishing an image-text cross-modal hash code library, characterized in that: include: For a data set consisting of mutually matching image-text pairs, the image-text cross-modal hash code extraction method described in claim 6 or 7 is used to extract the discrete hash code of each image to obtain an image hash code library, and the image-text cross-modal hash code extraction method described in claim 6 or 7 is used to extract the discrete hash code of each text to obtain a text hash code library.

9. An image-text cross-modal retrieval method, characterized in that: include: (T1) For the target object to be retrieved, determine its modality. If it is an image, proceed to (T2); if it is a text, proceed to (T3); (T2) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method described in claim 6 or 7, respectively calculate the similarity between the discrete hash code and each text hash code in the text hash code library, and select the text corresponding to the topK text hash codes with the highest similarity as the retrieval result, and proceed to (T4); (T3) After extracting the discrete hash code of the target object using the image-text cross-modal hash code extraction method described in claim 6 or 7, respectively calculate the similarity between the discrete hash code and the hash code of each image in the image hash code library, and select the images corresponding to the topK image hash codes with the highest similarity as the retrieval results, and proceed to (T4); (T4) Search ends; The image hash code library and the text hash code library are established by the image-text cross-modal hash code library establishment method according to claim 8; topK is a preset positive integer.

10. A computer-readable storage medium, characterized in that The invention comprises a stored computer program; when the computer program is executed by a processor, the method for establishing an image-text cross-modal hash code extraction model according to any one of claims 1 to 5, or the method for extracting an image-text cross-modal hash code according to claim 6 or 7, or the method for establishing an image-text cross-modal hash code library according to claim 8, or the method for retrieval of an image-text cross-modal image according to claim 9 is implemented.

Images