An unsupervised cross-modal retrieval method based on memory reserve hash

Abstract

The application provides an unsupervised cross-modal retrieval method based on memory reservation hash, relates to the technical field of unsupervised cross-modal retrieval, and comprises the following steps: feature extraction and hash learning; feature fusion and similarity matrix construction; model training and optimization. The application provides stable training supervision by using a historical memory similarity matrix construction module, and effectively improves the discrimination ability and retrieval robustness of the hash code. The method of the application does not need semantic labels, is suitable for unlabeled multimedia data retrieval in a real scene, has a light model structure, high retrieval efficiency, and has strong practical application ability and expansibility.

Description

Technical Field

[0001] This invention relates to the technical field of unsupervised cross-modal retrieval, and more particularly to an unsupervised cross-modal retrieval method based on memory-preserving hashing. Background Technology

[0002] Early cross-modal retrieval methods [1] Similarity calculations are primarily based on real-valued space mapping, such as canonical correlation analysis and cross-modal factor analysis, which map data from different modalities to a common real-valued space. While these methods can achieve semantic alignment between modalities, they face high storage and computational complexity when dealing with large-scale data. To address this challenge, cross-modal hashing methods have emerged, significantly improving retrieval efficiency by mapping multimodal data to a compact binary hash space.

[0003] Cross-modal hashing methods can be divided into two categories: supervised and unsupervised. Supervised methods include deep cross-modal hashing (DCMH). [2] Using semantic labels to guide hash code learning typically achieves good retrieval performance, but it relies on large-scale labeled data, resulting in high labeling costs and limiting its practical application. Unsupervised methods, on the other hand, avoid this dependence on labels and learn hash codes by mining the cross-modal correlations within the data itself, making them more practical. A representative method is Deep Joint Semantic Reconstruction Hash (DJSRH). [3] Semantic relationships are learned by constructing a joint semantic affinity matrix for cross-modal instances; Depth Graph Neighborhood Consistency Preservation Network (DGCPN) [4] This leads to the introduction of higher-order neighborhood structures to enhance semantic modeling capabilities. However, existing unsupervised cross-modal hashing methods still face two major challenges: first, insufficient feature interaction between modalities leads to misalignment of feature spaces; second, similarity matrices are usually constructed based on rapidly changing instantaneous features during training, lacking stability and easily introducing noise, which can cause fluctuations in the training process.

[0004] In the field of representation learning, Momentum Comparison Learning (MoCo) [5] The emergence of methods such as momentum update has provided new ideas for unsupervised feature learning. These methods maintain a historical feature memory bank through momentum update mechanisms, achieving stable and consistent feature representations and effectively alleviating the feature inconsistency problem during training. These advances provide valuable insights for constructing stable supervision signals in cross-modal hashing. However, existing methods have not fully integrated momentum memory mechanisms with cross-modal hashing, nor have they systematically addressed the collaborative optimization problem of cross-modal feature fusion and stable similarity supervision.

[0005] Most existing methods still employ a dual-encoder architecture to independently learn features from each modality, lacking deep cross-modal semantic interaction. Furthermore, the construction of similarity matrices relies on manually designed fusion strategies and instantaneous features, making it difficult to adapt to dynamic training processes and limiting further improvements in the discriminative power and retrieval performance of hash codes. Therefore, there is an urgent need for an unsupervised hashing method that can achieve deep cross-modal semantic fusion and provide stable similarity supervision to improve the accuracy and robustness of cross-modal retrieval.

[0006] References

[0007] [1] Andrew G, Arora R, Bilmes J, et al. Deep canonical correlation analysis[C] / / Proceedings of the 30th International Conference on MachineLearning. Atlanta, Georgia, USA: PMLR, 2013: 1247-1255.

[0008] [2] Jiang QY, Li W J. Deep cross-modal hashing[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 3232-3240.

[0009] [3] Su S, Zhong Z, Zhang C. Deep joint-semantics reconstructinghashing for large-scale unsupervised cross-modal retrieval[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 3027-3035.

[0010] [4] Yu J, Zhou H, Zhan Y, et al. Deep graph-neighbor coherencepreserving network for unsupervised cross-modal hashing[C] / / Proceedings of the AAAI conference on artificial intelligence. 2021, 35(5): 4626-4634.

[0011] [5] He K, Fan H, Wu Y, et al. Momentum contrast for unsupervised visual representation learning[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition. 2020: 9729-9738. Summary of the Invention

[0012] Traditional cross-modal retrieval methods typically map different modalities to a real-valued common space for similarity calculation. While this captures semantic relationships, it faces high storage and computational costs when handling large-scale data. This invention proposes an unsupervised cross-modal retrieval method based on memory-preserving hashing. This invention utilizes a historical memory similarity matrix construction module to provide stable training supervision, effectively improving the discriminative power and retrieval robustness of hash codes. The method requires no semantic labels, making it suitable for unlabeled multimedia data retrieval in real-world scenarios. Furthermore, the model structure is lightweight, retrieval efficiency is high, and it possesses strong practical application capabilities and scalability.

[0013] The technical means employed in this invention are as follows:

[0014] An unsupervised cross-modal retrieval method based on memory-preserving hashing includes the following steps:

[0015] Step 1: Feature Extraction and Hash Learning; A pre-trained encoder is used to extract the original semantic features of each image modality. Original semantic features of text modality The original semantic features of the image modality are obtained through a residual multilayer perceptron. Original semantic features of text modality Projecting onto the modality-shared information space yields the shared semantic features of the image modalities. Shared semantic features with text modality ; utilizing with A single-layer linear hashing layer of activation function will share the semantic features of the image modalities. Shared semantic features with text modality Compress to the corresponding dimensional relaxation real-valued hash features Using symbolic functions Generate the corresponding binary hash code ;

[0016] Step 2, Feature Fusion and Similarity Matrix Construction: The shared semantic features of the image modalities are fused using a cross-modal shared feature fusion module. Shared semantic features with text modality Adaptive fusion is performed to obtain unified cross-modal features. A historical memory bank is constructed based on the momentum update mechanism to store the cross-modal features. Historical information Combining cross-modal features With the historical information Constructing a cross-modal fusion semantic similarity matrix ;

[0017] Step 3: Model training and optimization; based on the real-valued hash features and cross-modal fusion semantic similarity matrix Construct a loss that preserves semantic similarity. Pairwise semantic similarity preservation loss Intramodal and intermodal similarity preservation loss Total training loss The model parameters are optimized by minimizing the total training loss, and a binary hash code is output.

[0018] Furthermore, the pre-trained encoder includes an image encoder and a text encoder; the image encoder is a Vision Transformer; and the text encoder is a GPT-2.

[0019] Furthermore, the residual multilayer perceptron includes The same MLP layer with residual connections.

[0020] Furthermore, in step 2, the shared semantic features are adaptively fused using a cross-modal shared feature fusion module to obtain unified cross-modal features. This includes the following steps:

[0021] Step 21: Shared semantic features of image modalities Shared semantic features with text modality The concat operation is used to concatenate the input features into a fused input feature. ,Right now:

[0022] ;

[0023] in, This indicates a feature splicing operation.

[0024] Step 22, through The Transformer encoder of the layer applies the fused input features Perform semantic fusion to generate unified cross-modal features The formula for the fusion process is: ;in, The parameter is Transformer encoder.

[0025] Furthermore, the momentum update mechanism of the historical memory bank satisfies:

[0026] ;

[0027] in, , Indicates the momentum update coefficient; Indicates the first in the memory bank Historical characteristics of each location; Indicates the number of iterations in the current iteration. For each image and text corresponding to a cross-modal feature, a mini-batch update strategy is adopted, where each iteration only updates the memory position corresponding to the current mini-batch sample.

[0028] Furthermore, the momentum update coefficients adopt a scaling momentum function that increases linearly with the number of iterations, satisfying: ;in, Indicates the current training iteration number; Both represent the hyperparameter for adjusting the momentum growth rate; This indicates the upper limit of the momentum coefficient.

[0029] Furthermore, the cross-modal features With the historical information Constructing a cross-modal fusion semantic similarity matrix The construction process is as follows: First, calculate the cross-modal features. With historical information cosine similarity Then through linear transformation The similarity values ​​are mapped to the target range to obtain the final cross-modal fusion semantic similarity matrix.

[0030] Furthermore, in step 3, the fusion semantic similarity preservation loss The calculation formula is:

[0031] ;

[0032] in, Indicates fused hash characteristics; Denotes the Frobenius norm of a matrix; This represents the cosine similarity between vectors.

[0033] Furthermore, in step 3, the pairwise semantic similarity preservation loss The calculation formula is:

[0034] ;

[0035] in, Represents the trace of a matrix; Represents the identity matrix; The hyperparameters representing the pairwise instance alignment adjustment are: intra-modal and inter-modal similarity preservation loss. The calculation formula is:

[0036] ;

[0037] in, , These represent image and text modalities, respectively.

[0038] Compared with the prior art, the present invention has the following advantages:

[0039] This invention takes a dual approach, considering both cross-modal semantic alignment and training stability, to accurately model the synergistic relationship between intermodal feature interactions and historical similarity supervision, thereby enhancing the discriminative power and retrieval robustness of hash codes. By employing a momentum memory mechanism to dynamically capture semantic evolution information, a stable cross-modal similarity guidance signal is constructed, exploring a broader cross-modal semantic representation space and achieving adaptive optimization of cross-modal hash learning in unsupervised scenarios. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0042] Figure 2 This is a schematic diagram illustrating the effect of constructing similarity matrices using traditional methods.

[0043] Figure 3 This is a schematic diagram illustrating the effect of similarity matrix construction according to the present invention. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0046] like Figure 1-3 As shown, this invention provides an unsupervised cross-modal retrieval method based on memory-preserving hashing, comprising the following steps:

[0047] Step 1: Feature Extraction and Hash Learning; A pre-trained encoder is used to extract the original semantic features of each image modality. Original semantic features of text modality This invention uses Vision Transformer and GPT-2 as image encoders and text encoders, respectively, and employs a residual multilayer perceptron to extract the original semantic features of the image modality. Original semantic features of text modality Projecting onto the modality-shared information space yields the shared semantic features of the image modalities. Shared semantic features with text modality ; utilizing with A single-layer linear hashing layer of activation function will share the semantic features of the image modalities. Shared semantic features with text modality Compress to the corresponding dimensional relaxation real-valued hash features Using symbolic functions Generate the corresponding binary hash code .

[0048] This can be understood as the rich feature representation in this application. ,in Subsequently, a residual multilayer perceptron (ResMLP) is introduced to project the features learned by the pre-trained model onto the modality shared information space, in order to learn commonalities and bridge the distribution gaps between different modalities.

[0049]

[0050] ResMLP includes The same MLP layer with residual connections, These are trainable parameters.

[0051] Subsequently, the present invention employs a device with A single-layer linear network with activation functions is used as a hash layer to compress high-dimensional shared semantic information representations. Features of dimensional relaxed real-valued hashing:

[0052]

[0053] in These are trainable parameters. Finally, using... The function generates strict binary hash codes:

[0054]

[0055] Step 2: Feature fusion and similarity matrix construction.

[0056] The focus of this invention is to model cross-modal semantic similarity relationships by exploring the latent semantic information between multimodal data in unsupervised scenarios. To establish effective and stable supervision for hash code learning, we introduce two core components: a cross-modal shared feature fusion module and a historical memory similarity matrix construction module. The cross-modal shared feature fusion module aims to learn a unified feature representation by adaptively fusing semantic information from different modalities. Based on these fused representations, the historical memory similarity matrix construction module employs a momentum update mechanism to maintain historical feature information and constructs a robust cross-modal semantic similarity matrix that remains consistent throughout the training process.

[0057] The cross-modal shared feature fusion module aims to fuse shared semantic features from different modalities to capture latent semantic relationships between multimodal data and bridge the heterogeneity gap between modalities. The cross-modal shared feature fusion module is used to fuse shared semantic features from the image modalities. Shared semantic features with text modality Adaptive fusion is performed to obtain unified cross-modal features. A historical memory bank is constructed based on the momentum update mechanism to store the cross-modal features. Historical information Combining cross-modal features With the historical information Constructing a cross-modal fusion semantic similarity matrix .

[0058] Specifically, in step 2, the shared semantic features are adaptively fused using a cross-modal shared feature fusion module to obtain unified cross-modal features. This includes the following steps:

[0059] Step 21: The cross-modal shared feature fusion module aims to fuse shared semantic features from different modalities to capture latent semantic relationships between multimodal data and bridge the heterogeneity gap between modalities. This involves fusing shared semantic features from image modalities. Shared semantic features with text modality The concat operation is used to concatenate the input features into a fused input feature. ,Right now:

[0060] ;

[0061] in, This indicates a feature splicing operation.

[0062] Step 22, through The Transformer encoder of the layer applies the fused input features Perform semantic fusion to generate unified cross-modal features The formula for the fusion process is: ;in, The parameter is The Transformer encoder enables the cross-modal shared feature fusion module to obtain a unified, representative, and comprehensive representation for paired instances, thereby mitigating the semantic gap between different modalities.

[0063] The Historical Memory Similarity Matrix Construction Module aims to build a robust cross-modal semantic similarity matrix by using historical feature information updated based on momentum.

[0064] In a preferred embodiment, in this application, in order to retain all valuable historical information generated by the cross-modal shared feature fusion module during model optimization, the construction of the historical memory bank based on the momentum update mechanism satisfies the following:

[0065] ;

[0066] in, , Indicates the momentum update coefficient; Indicates the first in the memory bank Historical characteristics of each location; Indicates the number of iterations in the current iteration. For each image and text corresponding to a cross-modal feature, a mini-batch update strategy is adopted, where each iteration only updates the memory position corresponding to the current mini-batch sample.

[0067] The momentum update coefficients adopt a scaling momentum function that increases linearly with the number of iterations, satisfying: ;in, Indicates the current training iteration number; Both represent the hyperparameter for adjusting the momentum growth rate; This indicates the upper limit of the momentum coefficient.

[0068] The cross-modal features With the historical information Constructing a cross-modal fusion semantic similarity matrix The construction process is as follows: First, calculate the cross-modal features. With historical information cosine similarity Then through linear transformation The similarity values ​​are mapped to the target range to obtain the final cross-modal fusion semantic similarity matrix.

[0069] For the momentum coefficient, unlike other methods that use a fixed, large momentum coefficient to ensure smooth updates, we employ a scaling momentum function that makes the momentum value increase linearly with the number of iterations, eventually stabilizing at a constant:

[0070]

[0071] in Indicates the first iteration It is to adjust the momentum coefficient Hyperparameters of growth rate This is the upper limit. In the initial iterations, we set a small momentum to encourage the memory to capture more cross-modal shared semantic information. As the number of iterations increases, the momentum gradually approaches 1, causing the memory updates to stabilize, ensuring the consistency of shared semantic representations, and preventing the degradation of inter-instance similarity modeling.

[0072] Furthermore, the goal is to project multimodal data onto a shared discrete... The process inevitably leads to a loss of precision, as the generated hash code may not retain the structural information embedded in the high-dimensional semantic features. Step 3: Model training and optimization; based on the real-valued hash features... and cross-modal fusion semantic similarity matrix Construct a loss that preserves semantic similarity. Pairwise semantic similarity preservation loss Intramodal and intermodal similarity preservation loss Total training loss The model parameters are optimized by minimizing the total training loss, and a binary hash code is output.

[0073] In step 3, the semantic similarity preservation loss is fused. The calculation formula is:

[0074] ;

[0075] in, Indicates fused hash characteristics; Denotes the Frobenius norm of a matrix; This represents the cosine similarity between vectors.

[0076] Furthermore, in step 3, the pairwise semantic similarity preservation loss The calculation formula is:

[0077] ;

[0078] in, Represents the trace of a matrix; Represents the identity matrix; The hyperparameters representing the pairwise instance alignment adjustment are: intra-modal and inter-modal similarity preservation loss. The calculation formula is:

[0079] ;

[0080] in, , These represent image and text modalities, respectively.

[0081] Furthermore, traditional unsupervised cross-modal hashing methods typically employ a dual-encoder architecture to independently learn features for each modality and construct a similarity matrix based on instantaneous features during training. For example... Figure 2 As shown, during the training iteration process, due to the rapid changes in feature representation, the similarity relationship fluctuates drastically, causing semantically similar samples (such as blue circles and yellow triangles) to gradually separate in the feature space, while semantically dissimilar samples may be misaligned. This unstable similarity supervision signal will seriously affect the learning quality of hash codes and reduce retrieval accuracy.

[0082] To address the aforementioned problems, this invention proposes an unsupervised cross-modal retrieval method based on memory-preserving hashing. This method achieves deep semantic alignment of image and text features through a cross-modal shared feature fusion module, effectively mitigating the semantic gap between modalities; and introduces a momentum-updated memory bank through a historical memory similarity matrix construction module to retain historical semantic information during training, constructing a stable and consistent similarity supervision signal. Figure 3 As shown, the MPH method can maintain the stability of semantic relationships during training iterations. For example, for a "zoo" scene (blue circle) that is semantically related to both "animal" (yellow triangle) and "natural landscape" (green square), MPH can maintain its similarity to both in continuous iterations, while traditional methods may cause semantic relationship breaks due to feature fluctuations.

[0083] Experiments demonstrate that this method effectively improves the accuracy and robustness of cross-modal retrieval while maintaining high retrieval efficiency. Furthermore, this method requires no manual annotation, making it suitable for retrieving massive amounts of unlabeled multimedia data in real-world scenarios, and exhibits good practicality and scalability.

[0084] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the above embodiments of the present invention, the descriptions of each embodiment have their own emphasis; parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the disclosed technical content in the several embodiments provided in this application can be implemented in other ways.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An unsupervised cross-modal retrieval method based on memory-preserving hashing, characterized in that, Includes the following steps: Step 1: Feature Extraction and Hash Learning; A pre-trained encoder is used to extract the original semantic features of each image modality. Original semantic features of text modality The original semantic features of the image modality are obtained through a residual multilayer perceptron. Original semantic features of text modality Projecting onto the modality-shared information space yields the shared semantic features of the image modalities. Shared semantic features with text modality ; utilizing with A single-layer linear hashing layer of activation function will share the semantic features of the image modalities. Shared semantic features with text modality Compress to the corresponding dimensional relaxation real-valued hash features Using symbolic functions Generate the corresponding binary hash code ; Step 2, Feature Fusion and Similarity Matrix Construction: The shared semantic features of the image modalities are fused using a cross-modal shared feature fusion module. Shared semantic features with text modality Adaptive fusion is performed to obtain unified cross-modal features. A historical memory bank is constructed based on the momentum update mechanism to store the cross-modal features. Historical information Combining cross-modal features With the historical information Constructing a cross-modal fusion semantic similarity matrix ; Step 3: Model training and optimization; based on the real-valued hash features and cross-modal fusion semantic similarity matrix Construct a loss that preserves semantic similarity. Pairwise semantic similarity preservation loss Intramodal and intermodal similarity preservation loss Total training loss The model parameters are optimized by minimizing the total training loss, and a binary hash code is output.

2. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, The pre-trained encoder includes an image encoder and a text encoder; the image encoder is VisionTransformer; and the text encoder is GPT-2.

3. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, The residual multilayer perceptron includes The same MLP layer with residual connections.

4. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, In step 2, the shared semantic features are adaptively fused using a cross-modal shared feature fusion module to obtain unified cross-modal features. This includes the following steps: Step 21: Shared semantic features of image modalities Shared semantic features with text modality The concat operation is used to concatenate the input features into a fused input feature. ,Right now: ； in, Indicates feature concatenation operation; Step 22, through The Transformer encoder of the layer applies the fused input features Perform semantic fusion to generate unified cross-modal features The formula for the semantic fusion process is: ;in, The parameter is Transformer encoder.

5. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, The construction of the historical memory bank based on the momentum update mechanism satisfies: ； in, , Indicates the momentum update coefficient; Indicates the first in the memory bank Historical characteristics of each location; Indicates the number of iterations in the current iteration. For each image and text corresponding to a cross-modal feature, a mini-batch update strategy is adopted, where each iteration only updates the memory position corresponding to the current mini-batch sample.

6. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 5, characterized in that, The momentum update coefficients adopt a scaling momentum function that increases linearly with the number of iterations, satisfying: ;in, Indicates the current training iteration number; Both represent the hyperparameter for adjusting the momentum growth rate; This indicates the upper limit of the momentum coefficient.

7. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, The cross-modal features With the historical information Constructing a cross-modal fusion semantic similarity matrix The construction process is as follows: First, calculate the cross-modal features. With historical information cosine similarity Then through linear transformation The similarity values ​​are mapped to the target range to obtain the final cross-modal fusion semantic similarity matrix.

8. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, In step 3, the semantic similarity preservation loss is fused. The calculation formula is: ； in, Indicates fused hash characteristics; Denotes the Frobenius norm of a matrix; This represents the cosine similarity between vectors.

9. The unsupervised cross-modal retrieval method based on memory-preserving hashing according to claim 1, characterized in that, In step 3, the pairwise semantic similarity preservation loss The calculation formula is: ； in, Represents the trace of a matrix; Represents the identity matrix; The hyperparameters representing the pairwise instance alignment adjustment are: intra-modal and inter-modal similarity preservation loss. The calculation formula is: ； in, , These represent image and text modalities, respectively.

Images