Incomplete cross-modal hashing retrieval method based on prediction completion and auxiliary code guidance

Abstract

This invention discloses an incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance, belonging to the field of cross-modal hash retrieval technology. This invention employs a pre-trained CLIP model as a cross-modal encoder to extract semantic features from image and text modalities, effectively enhancing the semantic consistency of cross-modal data. A bidirectional predictive network is designed, combined with a variational inference mechanism, to efficiently complete missing modal features using available modal features, successfully addressing the problem of modal loss caused by data acquisition failures or transmission interruptions. A residual contrastive network is introduced to improve the discriminative ability of the completed multimodal features and achieve feature alignment, effectively mitigating the distribution shift caused by missing data. In the hash encoding stage, a high-bit auxiliary code is used as a semantic teacher, guiding the low-bit hash code to learn richer semantic representations through knowledge distillation, significantly reducing information loss during compression. This invention significantly improves the efficiency and accuracy of incomplete cross-modal retrieval.

Description

Technical Field

[0001] This invention relates to an incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance, belonging to the field of cross-modal hash retrieval technology. Background Technology

[0002] With the rapid development of internet technology and the explosive growth of multimodal data, cross-modal retrieval has become one of the core tasks in the field of information retrieval. Its goal is to establish semantic relationships between different modalities of data (such as images and text) and achieve efficient matching of heterogeneous data. Deep Cross-Modal Hashing (DCMH) is widely used in large-scale cross-modal retrieval scenarios due to its advantages of low storage overhead and fast retrieval speed. Its core idea is to map high-dimensional multimodal features into compact binary hash codes and achieve fast similarity calculation through bitwise operations in Hamming space. In recent years, researchers have proposed a series of deep cross-modal hashing methods, such as supervised learning-based DCMH and PRDH, unsupervised learning-based DJSRH and UCMFH, and pre-trained model-based DCHMT and PGCH. These methods have achieved significant performance improvements in complete multimodal data scenarios.

[0003] However, existing deep cross-modal hashing methods still face two major challenges: First, most methods rely on the ideal assumption of "complete and available multimodal data," ignoring the problem of modality loss caused by data acquisition failures, transmission interruptions, and device limitations in practical applications. This leads to a significant drop in retrieval performance in scenarios with incomplete data. Second, to adapt to resource-constrained environments such as edge devices, low-bit hash codes (such as 16-bit and 32-bit) are widely used, but severe information loss occurs during compression, limiting the semantic representation ability of the hash code and making it difficult to retain fine-grained semantic information. To address these issues, some researchers have attempted related improvements, but significant limitations remain. Completion methods based on Generative Adversarial Networks (GANs) suffer from training instability and semantic bias in generated features; while completion methods based on prototype clustering heavily rely on the quality of prototype generation and lack robustness in scenarios with high missing rates. Regarding low-bit encoding optimization, existing methods mostly focus on adjusting the hash loss function or improving cross-modal alignment strategies, but lack enhancement designs for the semantic carrying capacity of the hash code itself, failing to effectively compensate for the information loss defects of low-bit encoding. Furthermore, most existing studies treat modality missing fit and low-bit encoding optimization as two independent problems to be solved separately, lacking a collaborative design approach. This makes it difficult for models to simultaneously address both challenges, further limiting their application value in real-world scenarios. Therefore, developing a cross-modal hash retrieval method that can collaboratively solve the problems of modality missing fit and low-bit encoding information loss, while also possessing high retrieval accuracy, high efficiency, and strong robustness, has significant practical application value. Summary of the Invention

[0004] In view of the shortcomings of existing technologies, this invention proposes an incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance. Through the collaborative design of bidirectional predictive completion, residual comparison alignment and high-bit auxiliary code distillation, the technical problems of modality loss and low-bit encoded information loss are solved.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: an incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance, comprising the following steps:

[0006] Step 1: Data Acquisition: Collect incomplete cross-modal datasets;

[0007] Step 2, Data Processing: First, the collected dataset is preprocessed, with zero-padding performed on samples with missing modalities to obtain standardized input data. Then, the dataset is divided into non-overlapping training sample sets, query sample sets, and retrieval sample sets.

[0008] Step 3, Multimodal Feature Extraction: Construct a feature extraction module, using a pre-trained CLIP model as a modality-specific encoder to encode the preprocessed image and text input respectively, and extract high-dimensional semantic features;

[0009] Step 4, Cross-modal completion and alignment: Construct a cross-modal completion and alignment module. First, use a bidirectional prediction network to complete the missing modal features using available modal features to obtain a complete multimodal feature matrix. Then, use a residual contrastive network to perform nonlinear optimization on the completed features and achieve cross-modal feature alignment through contrastive loss.

[0010] Step 5, Hash Code Learning: Construct a dual-branch hash coding network to generate low-bit hash codes and high-bit auxiliary codes for the aligned features respectively; use the high-bit auxiliary code as a semantic teacher to guide the learning of low-bit hash codes through knowledge distillation; at the same time, optimize the intra-modal similarity preservation loss, inter-modal similarity preservation loss and quantization loss to obtain the final hash coding model;

[0011] Step 6: Model Training: Train the incomplete cross-modal hash retrieval network guided by prediction completion and auxiliary code using the image-text training sample set. After each batch of training is completed, verify the trained network using the image-text query sample set to check the current status and convergence of the method.

[0012] Step 7, Incomplete cross-modal retrieval: Load the trained model, input the query sample set and the retrieval sample set, and generate the corresponding hash codes respectively; calculate the Hamming distance between the hash codes of the query samples and the retrieval samples, and sort them in ascending order of distance to obtain the retrieval results.

[0013] Furthermore, in Step 1, the dataset can be selected from publicly available benchmark datasets such as MIRFLICKR-25K, NUS-WIDE, and MS-COCO, or a custom dataset can be constructed according to the actual application scenario. Each sample in the dataset must contain at least 2 semantic categories, and the number of training samples should not be less than 10,000 to ensure the model training effect.

[0014] Furthermore, in Step 3, the CLIP model encoder adopts the Transformer architecture. The image encoder takes a standardized image of 224×224 pixels as input, and the text encoder takes a text sequence that has been segmented into words as input. After encoding, the output feature dimension d=512.

[0015] In Step 1, the incomplete cross-modal dataset satisfies: total number of samples ,in The number of complete image-text sample pairs. This represents the number of single-modal samples containing only images. The number of single-modal samples containing only text, with sample labels containing at least one semantic category.

[0016] Furthermore, in Step 3, the CLIP model contains two structurally independent image encoders and text encoders, which can encode image and text modal data into image and text features, respectively.

[0017] Furthermore, the bidirectional prediction network in Step 4 adopts an MLP structure with a dimension configuration of 512→2048→512 and a mask matrix. , Dynamically generated based on sample modality integrity (complete sample mask value is 1, missing sample mask value is 0); the bidirectional prediction network in Step 4 consists of two modality prediction networks each constructed from a two-layer multilayer perceptron; assuming modality features X and Y follow an implicit joint distribution. In the incomplete modal setting, the prediction completion module aims to maximize the conditional likelihood. To infer from the observed modes For the missing mode of conditions , It adopts a variational perspective of conditional density estimation and introduces a tractable variational distribution. To approximate the target conditional distribution; further assume that the variational distribution follows an isotropic Gaussian distribution: To simplify the calculations, the final objective function is simplified to:

[0018] in, This represents matrix multiplication, where M represents the size of the current batch, and X and Y represent image and text features, respectively. , This represents the feature of the i-th image and text sample in the current batch, where i is the sample index in the current batch; It is a modal identifier used to distinguish between image and text modalities. and These represent image-to-text and text-to-image prediction networks, respectively. and These represent the missing mask matrices for the image and text modalities, respectively.

[0019] Furthermore, the residual contrast network in Step 4 employs a ResMLP consisting of two layers of perceptron connected to residuals, with GELU as the activation function and a temperature hyperparameter τ=0.07 for the contrast loss. A two-layer residual multilayer perceptron with weight sharing is used to complete the image text features. , Nonlinear enhancement is performed to obtain enhanced image text features. and Finally, cross-modal alignment is achieved through symmetric contrastive loss, with the contrastive alignment loss function being:

[0020]

[0021] in, This is the temperature hyperparameter, where M represents the batch size. These are modal identifiers used to distinguish between image and text modalities. i and j are sample indices within the current batch. , Let these represent the image features and text features of the i-th sample in the current batch, respectively. , These represent the image features and text features of the j-th sample in the current batch, respectively.

[0022] Furthermore, in Step 5, the low-bit hash code length The hash encoding network can be selected as 16, 32, or 64 bits. It employs a linear projection layer combined with a Tanh activation function to ensure the continuity of output features for gradient descent optimization. The decoder uses an MLP structure with a dimensionality configuration of [missing information]. → → The low-bit hash code and high-bit auxiliary code are generated by a hash network sharing their respective modes, resulting in the low-bit space image feature embedding. Text feature embedding Image feature embedding in high bit space Text feature embedding The final hash code needs to be transformed by the sign function; the hash network consists of a linear layer plus the tanh activation function.

[0023] Furthermore, in Step 5,

[0024] (1) Intramodal similarity preservation loss:

[0025]

[0026] Where N represents the total number of samples in the dataset, M represents the size of the current batch, k represents the sample index of the dataset, and j represents the sample index within the current batch; This represents the similarity between the k-th sample in the dataset and the j-th sample in the current batch. The inner product represents the feature embeddings of the low-bit space within a modality. , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This represents the low-bit space image feature embedding of the k-th sample in the dataset. This represents the low-bit space image feature embedding of the j-th sample in the current batch, where the value of "*" is... hour, This represents the text feature embedding in the low-bit space of the k-th sample in the dataset. This represents the text feature embedding in the low-bit space of the j-th sample within the current batch;

[0027] (2) Intermodal similarity preservation loss:

[0028]

[0029] in, and The inner product represents the feature embeddings in the low-bit space between modalities. It is a modal identifier used to distinguish between image and text modalities. , These represent the low-bit space image and text feature embeddings of the k-th sample in the dataset, respectively. , These represent the low-bit space image and text feature embeddings of the j-th sample in the current batch, respectively.

[0030] (3) Quantification of loss:

[0031]

[0032] Where K is the hash code length, and the unified hash code is... This represents the unique hash code of the i-th sample in the current batch. , Let represent the low-bit space image and text feature embeddings of the i-th sample in the current batch, respectively, and let sign denote the sign function. , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This represents the low-bit space image feature embedding of the i-th sample in the current batch, where the value of "*" is... hour, This represents the text feature embedding in the low-bit space of the i-th sample within the current batch; the semantic preservation loss of the low-bit hash code is finally obtained.

[0033]

[0034] in and This represents the weight hyperparameter;

[0035] Furthermore, in the above (1) intra-modal similarity preservation loss, (2) inter-modal similarity preservation loss, and (3) quantization loss, the low-bit space image and text feature embeddings used are replaced with high-bit space image and text feature embeddings. The intra-modal similarity preservation loss, inter-modal similarity preservation loss, and quantization loss are calculated again, and the semantic preservation loss of the auxiliary code is obtained by weighting them together. .

[0036] Furthermore, in Step 5, knowledge distillation maps the low-bit hash code to the auxiliary code semantic space through a decoder composed of two multilayer perceptrons, and minimizes the reconstruction loss:

[0037]

[0038] Where M represents the size of the current batch, , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This indicates that the i-th image modality sample in the current batch is obtained by decoding the high-bit space feature embedding obtained from the low-bit space feature embedding. The value of "*" is... hour, This indicates that the i-th text modality sample in the current batch is obtained by decoding the high-bit space feature embedding obtained from the low-bit space feature embedding. This represents the unified hash code of the i-th auxiliary code in the current batch.

[0039] Furthermore, the total loss function in Step 5 is:

[0040]

[0041] in, This represents the hyperparameters that balance the various components.

[0042] The beneficial effects of this invention are:

[0043] 1. This invention achieves efficient completion of missing modalities through bidirectional prediction networks and variational inference mechanisms. Compared with unidirectional completion strategies, it effectively avoids modal bias and improves the model's adaptability to different missing rate scenarios.

[0044] 2. The introduction of a residual contrast network performs nonlinear optimization and cross-modal alignment on the completed features, which alleviates the distribution shift caused by missing data and enhances the discriminative ability of the features;

[0045] 3. By using high-bit auxiliary codes and knowledge distillation mechanisms, the semantic representation defects of low-bit hash codes are effectively compensated, reducing the loss of compressed information while ensuring retrieval efficiency. Attached Figure Description

[0046] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention.

[0047] Figure 1 This is an overall flowchart of the method of the present invention;

[0048] Figure 2 This is a schematic diagram of the architecture of the method of the present invention;

[0049] Figure 3 This is a comparison chart of the PR curve and TopK-Precision curve of this invention with other cross-modal hashing methods. Detailed Implementation

[0050] The following descriptions are illustrative and are intended to further illustrate the technical solutions of the present invention in conjunction with the accompanying drawings.

[0051] Example 1

[0052] Figure 1 Figure 2 is a flowchart of an incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance according to the present invention. Figure 2 is a schematic diagram of the architecture of the present invention. This embodiment takes the MIRFLICKR-25K dataset as an example to explain the specific implementation process of the method in detail:

[0053] 1. Data Acquisition: The MIRFLICKR-25K dataset was used, consisting of 24,581 images collected from Flickr. Each image is accompanied by a corresponding text description and ground truth label. Each sample belongs to at least one of 24 categories. Incomplete datasets were constructed with three missing data percentages: Case 1 (10% complete samples, 45% image-only samples, 45% text-only samples), Case 2 (30% complete samples, 35% image-only samples, 35% text-only samples), and Case 3 (50% complete samples, 25% image-only samples, 25% text-only samples). Dataset Split: 10,000 samples for training, 2,000 samples for query set, and 22,581 samples for database.

[0054] 2. Data preprocessing: Zero-padding is performed on the text fields of image samples and zero-padding is performed on the image fields of text samples; trainable 2D positional codes are added to the padded image input and trainable 1D positional codes are added to the padded text input; image inputs are uniformly scaled to 224×224 pixels, and text inputs are processed by CLIP tokenizer into sequences with a length not exceeding 32.

[0055] 3. Multimodal feature extraction: Construct a feature extraction module, using a pre-trained CLIP model as a modality-specific encoder to encode the preprocessed image and text input respectively, and extract high-dimensional semantic features;

[0056] Specifically, a pre-trained CLIP-ViT-B / 32 model is used as the encoder. Both the image encoder and the text encoder are loaded with pre-trained weights and fine-tuned. The image encoder outputs 512-dimensional image features X, and the text encoder outputs 512-dimensional text features Y.

[0057] 4. Cross-modal completion and alignment: A cross-modal completion and alignment module is constructed. First, a bidirectional prediction network is used to complete the missing modal features using available modal features to obtain a complete multimodal feature matrix. Then, a residual contrastive network is used to perform nonlinear optimization on the completed features, and cross-modal feature alignment is achieved through contrastive loss.

[0058] Specifically, this includes:

[0059] 4.1 Mutual Prediction Completion: First, a bidirectional prediction network is used to complete the missing modal features using available modal features. The bidirectional prediction network adopts an MLP structure with a dimension configuration of 512→2048→512, and the mask matrix... , Dynamically generated based on sample modal integrity (mask value of 1 for complete samples, mask value of 0 for missing samples). Assume modal features X and Y follow an implicit joint distribution. In the incomplete modal setting, the predictive completion module aims to maximize the conditional likelihood. To infer from the observed modes For the missing mode of conditions , However, due to the unknown joint distribution, directly modeling this conditional distribution is difficult. Therefore, we adopt a variational perspective of conditional density estimation and introduce a tractable variational distribution. To approximate the target conditional distribution; further assume that the variational distribution follows an isotropic Gaussian distribution: To simplify the calculations, the final objective function is simplified to:

[0060] in, This represents matrix multiplication, where M represents the size of the current batch, and X and Y represent image and text features, respectively. , This represents the feature of the i-th image and text sample in the current batch, where i is the sample index in the current batch; It is a modal identifier used to distinguish between image and text modalities. and These represent image-to-text and text-to-image prediction networks, respectively. and These represent the missing mask matrices for the image and text modalities, respectively.

[0061] 4.2 Residual Contrast Alignment Learning: After obtaining the complete multimodal feature matrix, a residual contrastive network is used to perform nonlinear optimization on the completed features, achieving cross-modal feature alignment through contrastive loss. The residual contrastive network uses ResMLP, which includes two layers of perceptron connected to residuals. The activation function is GELU, and the temperature hyperparameter τ=0.07 for the contrastive loss. A two-layer residual multilayer perceptron with weight sharing is used to complete the image and text features. , Nonlinear enhancement is performed to obtain enhanced image text features. and Finally, cross-modal alignment is achieved through symmetric contrastive loss, with the contrastive alignment loss function being:

[0062]

[0063] in, This is the temperature hyperparameter, where M represents the batch size. These are modal identifiers used to distinguish between image and text modalities. i and j are sample indices within the current batch. , Let these represent the image features and text features of the i-th sample in the current batch, respectively. , These represent the image features and text features of the j-th sample in the current batch, respectively.

[0064] 5. Learning hash codes:

[0065] 5.1 Low-bit hash code length 16, 32, or 64 bits are selectable; the hash encoding branch is a linear layer (512→). Tanh activation is used, and the auxiliary coding branch is a linear layer (512→128) + Tanh activation. The low-bit hash code and high-bit auxiliary code are generated by a hash network sharing their respective modes, resulting in the low-bit space image feature embedding. Text feature embedding Image feature embedding in high bit space Text feature embedding The final hash code needs to be transformed by the sign function; the hash network consists of a linear layer plus the tanh activation function.

[0066] 5.2 Optimize intra-modal similarity preservation loss, inter-modal similarity preservation loss, and quantization loss:

[0067] (1) Intramodal similarity preservation loss:

[0068]

[0069] Where N represents the total number of samples in the dataset, M represents the size of the current batch, k represents the sample index of the dataset, and j represents the sample index within the current batch; This represents the similarity between the k-th sample in the dataset and the j-th sample in the current batch. The inner product represents the feature embeddings of the low-bit space within a modality. , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This represents the low-bit space image feature embedding of the k-th sample in the dataset. This represents the low-bit space image feature embedding of the j-th sample in the current batch, where the value of "*" is... hour, This represents the text feature embedding in the low-bit space of the k-th sample in the dataset. This represents the text feature embedding in the low-bit space of the j-th sample within the current batch;

[0070] (2) Intermodal similarity preservation loss:

[0071]

[0072] in, and The inner product represents the feature embeddings in the low-bit space between modalities. It is a modal identifier used to distinguish between image and text modalities. , These represent the low-bit space image and text feature embeddings of the k-th sample in the dataset, respectively. , These represent the low-bit space image and text feature embeddings for the j-th sample in the current batch, respectively.

[0073] (3) Quantification of loss:

[0074]

[0075] Where K is the hash code length, and the unified hash code is... This represents the unique hash code of the i-th sample in the current batch. , Let represent the low-bit space image and text feature embeddings of the i-th sample in the current batch, respectively, and let sign denote the sign function. , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This represents the low-bit space image feature embedding of the i-th sample in the current batch, where the value of "*" is... hour, This represents the text feature embedding in the low-bit space of the i-th sample within the current batch; the semantic preservation loss of the low-bit hash code is finally obtained.

[0076]

[0077] in =0.005 and =5 indicates the weight hyperparameter.

[0078] Furthermore, in the above (1) intra-modal similarity preservation loss, (2) inter-modal similarity preservation loss, and (3) quantization loss, the low-bit space image and text feature embeddings used are replaced with high-bit space image and text feature embeddings. The intra-modal similarity preservation loss, inter-modal similarity preservation loss, and quantization loss are calculated again, and the semantic preservation loss of the auxiliary code is obtained by weighting them together. .

[0079] 5.3 Knowledge distillation of low-bit hash codes from high-bit auxiliary codes: The decoder adopts an MLP structure with the following dimension configuration. → → In Step 5, knowledge distillation maps low-bit hash codes to auxiliary code semantic spaces using a decoder composed of two multilayer perceptrons, minimizing the reconstruction loss.

[0080]

[0081] Where M represents the size of the current batch, , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This indicates that the i-th image modality sample in the current batch is obtained by decoding the high-bit space feature embedding obtained from the low-bit space feature embedding. The value of "*" is... hour, This indicates that the i-th text modality sample in the current batch is obtained by decoding the high-bit space feature embedding obtained from the low-bit space feature embedding. This represents the unified hash code of the i-th auxiliary code in the current batch.

[0082] Furthermore, the total loss function in the incomplete cross-modal hash retrieval model based on prediction completion and auxiliary code guidance is:

[0083]

[0084] in, =5、 =5、 =0.5、 =0.001 indicates that the hyperparameters of each component are balanced.

[0085] 6. Model training: AdamW optimizer was used with a weight decay of 0.01; encoder learning rate was 1e-6, and other module learning rates were 2e-3; batch size was 128, and training iterations were 30.

[0086] 7. Incomplete cross-modal retrieval: Load the trained model, input the query sample (image or text), and generate a hash code after preprocessing, feature extraction, completion and alignment; calculate the Hamming distance between the query hash code and the hash codes of the database samples, sort them in ascending order of distance, and return the Top-K retrieval results; use mAP@all as the evaluation metric.

[0087] The hardware platform for the simulation experiment in this embodiment is: a 12th Gen Intel(R) Core(TM) i9-12900KF CPU and an NVIDIA GeForce RTX 4090 GPU with 24GB of video memory; the operating system is Ubuntu 20.04.5LTS, and the virtual environment configured includes: Python 3.8.3, PyTorch 1.10.0, CUDA 12.4, etc.

[0088] We feed the cross-modal query sample set into the incomplete cross-modal hash retrieval (PCEH-ACG) network trained above, and calculate the general evaluation metric: mean average precision (mAP). The larger the metric, the better the classification performance.

[0089] To evaluate the effectiveness of our method, we searched the MIRFLICKR-25K cross-modal dataset using nine state-of-the-art unsupervised cross-modal hashing methods: DJSRH, UCMFH, DSPH, GCIMH, DNPH, CICH, DDSS, DNcH, and DScPH.

[0090] The DJSRH method mentioned above refers to the deep joint semantic reconstruction hashing method for large-scale unsupervised cross-modal retrieval proposed by Su et al. in "Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval", abbreviated as DJSRH.

[0091] The UCMFH method mentioned refers to the unsupervised contrastive multimodal hashing retrieval method proposed by Xia et al. in "When CLIP meets cross-modal hashingretrieval: A new strong baseline", abbreviated as UCMFH;

[0092] The DSPH method mentioned above refers to the deep semantic-aware proxy hashing retrieval method for multi-label cross-modal retrieval proposed by Huo et al. in "Deep semantic-aware proxy hashing for multi-label cross-modal retrieval", abbreviated as DSPH;

[0093] The GCIMH method mentioned above refers to the graph convolutional incomplete multimodal hashing retrieval method proposed by Shen et al. in "Graph convolutional incomplete multimodal hashing", abbreviated as GCIMH;

[0094] The DNPH method mentioned above refers to the deep neighborhood-aware proxy hashing retrieval method with uniform distribution constraints proposed by Huo et al. in "Deep neighborhood-aware proxy hashing with uniform distribution constraint for cross-modal retrieval", abbreviated as DNPH;

[0095] The CICH method mentioned refers to the contrastive incomplete cross-modal hashing retrieval method proposed by Luo et al. in "Contrastive incomplete cross-modal hashing", abbreviated as CICH.

[0096] The DDSS method mentioned above refers to the dual-driven cross-modal contrastive hashing retrieval method based on structural features and semantic information proposed by Huang et al. in "Dual-driven cross-modal contrastive hashing retrieval network via structural feature and semantic information", abbreviated as DDSS.

[0097] The DNcH method mentioned refers to the deep neighbor-coherence hashing retrieval method based on discriminative sample mining proposed by Huo et al. in "Deep neighbor-coherence hashing with discriminative sample mining for supervised cross-modal retrieval", abbreviated as DNcH;

[0098] The DScPH method refers to the deep semantic consistency penalizing hash retrieval method for cross-modal retrieval proposed by Huo et al. in "Deep Semantic-consistent Penalizing Hashing for Cross-modal Retrieval", abbreviated as DScPH.

[0099] Table 1. mAP results for three incomplete scales on the MIRFLICKR-25K dataset.

[0100] As shown in Table 1, the proposed PCEH-ACG method achieves higher mAP values ​​than other comparative methods in both query tasks across three incompleteness scales on the MIRFLICKR-25K dataset. This further demonstrates the powerful performance of the proposed PCEH-ACG method in incomplete cross-modal retrieval.

[0101] Figure 3 This paper presents the PR and TopK-Precision curves of the proposed PCEH-ACG method on the MIRFLICKR-25K dataset with the incompleteness ratio of Case 2 (0.3, 0.35, 0.35) and a 32-bit hash code length. We can see that this method surpasses all compared methods in retrieval accuracy by retrieving more correct instances, demonstrating strong retrieval capabilities compared to other methods.

[0102] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance, characterized in that: The specific steps of the method are as follows: Step 1, Data Acquisition: Collect incomplete cross-modal datasets; Step 2, Data Processing: First, the collected dataset is preprocessed, and zero-padding is performed on samples with missing modalities to obtain standardized input data; then, the dataset is divided into non-overlapping training sample sets, query sample sets, and retrieval sample sets. Step 3, Multimodal Feature Extraction: Construct a feature extraction module, using a pre-trained CLIP model as a modality-specific encoder to encode the preprocessed image and text input respectively, and extract high-dimensional semantic features; Step 4, Cross-modal completion and alignment: Construct a cross-modal completion and alignment module. First, use a bidirectional prediction network to complete the missing modal features using available modal features to obtain a complete multimodal feature matrix. Then, use a residual contrastive network to perform nonlinear optimization on the completed features and achieve cross-modal feature alignment through contrastive loss. Step 5, Hash Code Learning: Construct a dual-branch hash coding network to generate low-bit hash codes and high-bit auxiliary codes for the aligned features respectively; Using high-bit auxiliary codes as semantic teachers, low-bit hash codes are guided to learn through knowledge distillation; at the same time, the intra-modal similarity preservation loss, inter-modal similarity preservation loss, and quantization loss are optimized to obtain the final hash coding model. Step 6: Model Training: Train the incomplete cross-modal hash retrieval network guided by prediction completion and auxiliary code using the image-text training sample set. After each batch of training is completed, verify the trained network using the image-text query sample set to check the current status and convergence of the method. Step 7, Incomplete cross-modal retrieval: Load the trained model, input the query sample set and the retrieval sample set, and generate the corresponding hash codes respectively; Calculate the Hamming distance between the hash codes of the query sample and the retrieved sample, and sort the results in ascending order of distance to obtain the retrieval results.

2. The incomplete cross-modal hash retrieval method based on prediction completion and auxiliary code guidance according to claim 1, characterized in that: In Step 1, the incomplete cross-modal dataset satisfies: total number of samples ,in The number of complete image-text sample pairs. This represents the number of single-modal samples containing only images. The number of single-modal samples containing only text, with sample labels containing at least one semantic category.

3. The incomplete cross-modal hash retrieval method based on prediction completion and auxiliary code guidance according to claim 1, characterized in that: In Step 3, the CLIP model contains two structurally independent image encoders and text encoders, which can encode image and text modal data into image and text features, respectively.

4. The incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance according to claim 1, characterized in that: In Step 4, the bidirectional prediction network consists of two modal prediction networks, each constructed from a two-layer multilayer perceptron; it is assumed that the modal features X and Y follow an implicit joint distribution. In the incomplete modal setting, the prediction completion module aims to maximize the conditional likelihood. To infer from the observed modes For the missing mode of conditions , It adopts a variational perspective of conditional density estimation and introduces a tractable variational distribution. To approximate the target conditional distribution; further assume that the variational distribution follows an isotropic Gaussian distribution: To simplify the calculations, the final objective function is simplified to: ; in, This represents matrix multiplication, where M represents the size of the current batch, and X and Y represent image and text features, respectively. , This represents the feature of the i-th image and text sample in the current batch, where i is the sample index in the current batch; It is a modal identifier used to distinguish between image and text modalities. and These represent image-to-text and text-to-image prediction networks, respectively. and These represent the missing mask matrices for the image and text modalities, respectively.

5. The incomplete cross-modal hash retrieval method based on prediction completion and auxiliary code guidance according to claim 1, characterized in that: The residual contrast network in Step 4 uses a two-layer residual multilayer perceptron with shared weights to complete the image text features. , Nonlinear enhancement is performed to obtain enhanced image text features. and Finally, cross-modal alignment is achieved through symmetric contrastive loss, with the contrastive alignment loss function being: ； in, This is the temperature hyperparameter, where M represents the batch size. These are modal identifiers used to distinguish between image and text modalities. i and j are sample indices within the current batch. , Let these represent the image features and text features of the i-th sample in the current batch, respectively. , These represent the image features and text features of the j-th sample in the current batch, respectively.

6. The incomplete cross-modal hash retrieval method based on prediction completion and auxiliary code guidance according to claim 1, characterized in that: In Step 5, the low-bit hash code and the high-bit auxiliary code are generated by their respective modal-shared hash networks to obtain the image feature embedding in the low-bit space. Text feature embedding Image feature embedding in high bit space Text feature embedding The final hash code needs to be transformed by the sign function; the hash network consists of a linear layer plus the tanh activation function.

7. The incomplete cross-modal hash retrieval method based on prediction completion and auxiliary code guidance according to claim 1, characterized in that: In Step 5, (1) Intramodal similarity preservation loss: ； Where N represents the total number of samples in the dataset, M represents the size of the current batch, k represents the sample index of the dataset, and j represents the sample index within the current batch; This represents the similarity between the k-th sample in the dataset and the j-th sample in the current batch. The inner product represents the feature embeddings of the low-bit space within a modality. , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This represents the low-bit space image feature embedding of the k-th sample in the dataset. This represents the low-bit space image feature embedding of the j-th sample in the current batch, where the value of "*" is... hour, This represents the text feature embedding in the low-bit space of the k-th sample in the dataset. This represents the text feature embedding in the low-bit space of the j-th sample within the current batch; (2) Intermodal similarity preservation loss: ； in, and The inner product represents the feature embeddings in the low-bit space between modalities. It is a modal identifier used to distinguish between image and text modalities. , These represent the low-bit space image and text feature embeddings of the k-th sample in the dataset, respectively. , These represent the low-bit space image and text feature embeddings of the j-th sample in the current batch, respectively. (3) Quantification of loss: ； Where K is the hash code length, and the unified hash code is... This represents the unique hash code of the i-th sample in the current batch. , Let represent the low-bit space image and text feature embeddings of the i-th sample in the current batch, respectively, and let sign denote the sign function. , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This represents the low-bit space image feature embedding of the i-th sample in the current batch, where the value of "*" is... hour, This represents the text feature embedding in the low-bit space of the i-th sample within the current batch; the semantic preservation loss of the low-bit hash code is finally obtained. ； in and This represents the weight hyperparameter; Furthermore, in the above (1) intra-modal similarity preservation loss, (2) inter-modal similarity preservation loss, and (3) quantization loss, the low-bit space image and text feature embeddings used are replaced with high-bit space image and text feature embeddings. The intra-modal similarity preservation loss, inter-modal similarity preservation loss, and quantization loss are calculated again, and the semantic preservation loss of the auxiliary code is obtained by weighting them together. .

8. The incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance according to claim 1, characterized in that: In Step 5, knowledge distillation maps low-bit hash codes to auxiliary code semantic spaces using a decoder composed of two multilayer perceptrons, minimizing the reconstruction loss. ； Where M represents the size of the current batch, , It is a modal identifier used to distinguish between image and text modalities. When the value of "*" is hour, This indicates that the i-th image modality sample in the current batch is obtained by decoding the high-bit space feature embedding obtained from the low-bit space feature embedding. The value of "*" is... hour, This indicates that the i-th text modality sample in the current batch is obtained by decoding the high-bit space feature embedding obtained from the low-bit space feature embedding. This represents the unified hash code of the i-th auxiliary code in the current batch.

9. The incomplete cross-modal hash retrieval method based on predictive completion and auxiliary code guidance according to claim 1, characterized in that: The total loss function in Step 5 is: ； in, This represents the hyperparameters that balance the various components.

Images