Semantic-guided discriminant and robust unsupervised cross-modal hashing method

Abstract

The invention discloses a semantic-guided discriminant and robust unsupervised cross-modal hashing method, which comprises the following steps of: acquiring a multi-modal training data set of texts and images to obtain a pre-processed batch training data set; constructing and initializing a multi-modal feature extractor and a soft symbol hash layer; training to obtain a cross-modal Hash model based on the multi-modal feature extractor and the soft symbol Hash layer; and taking a text or an image as input based on the cross-modal hash model, and retrieving in an image or text pool to obtain a semantic matching result. According to the invention, based on native semantic information, image and text multi-modal feature extraction is realized by using a pre-training encoder and an attention mechanism, and the model is endowed with comprehensive extraction of image and text fine-grained features; through guidance and supervision of a self-adaptive robust prototype supervised learning mechanism, a semantic center is used as a category center of the Hash codes to carry out unsupervised classification, so that classification discrimination of the Hash codes and robust Hash representation learning under a pseudo tag are realized, and cross-modal efficient retrieval is realized.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal hashing technology, specifically involving a semantically guided, discriminative, and robust unsupervised cross-modal hashing method. Background Technology

[0002] With the rapid development of the internet, smart devices, and sensor technologies, multimodal data is growing exponentially, encompassing various forms such as images, text, audio, and video. This data is often used to describe the same events or themes. This data diversity has created an urgent need for cross-modal data retrieval. By promoting semantic alignment and leveraging the synergy between multimodal data, interaction and semantic matching between different modalities have been achieved. Cross-modal hashing technology, especially unsupervised cross-modal hashing, has attracted much attention because it can overcome the storage and efficiency bottlenecks in large-scale multimodal data retrieval. Unsupervised cross-modal hashing does not rely on manually labeled categories, but only utilizes the co-occurrence information of the data itself, making it more valuable for practical applications.

[0003] Despite significant progress in existing unsupervised cross-modal hashing techniques, there are still many shortcomings in achieving hash learning that combines high discriminative power and strong robustness, as detailed below:

[0004] Prototype learning mechanisms are fragile and prone to getting trapped in local optima: the initial values ​​of the prototypes (cluster centers) have a decisive impact on the performance of the final model. Poor initialization can easily cause the entire model to quickly fall into local optima, making the semantic clusters unable to accurately reflect the true data distribution. Most methods use a simple averaging method (i.e., averaging the embeddings of all samples within a cluster) to update the prototype. This strategy makes the prototype calculation highly susceptible to interference from outliers and noisy samples, causing the prototype to "drift" and fail to stably represent the semantic center of the category. This fragility makes the supervision signal provided by the prototype itself unstable, thus limiting the improvement of model robustness.

[0005] Pseudo-label supervision is unreliable, leading to poor model training robustness: Pseudo-label generation methods face significant risks of error accumulation and propagation. The quality of generated pseudo-labels is highly dependent on the quality of initial features and the effectiveness of the clustering algorithm. Under unsupervised conditions, unavoidable noise in the initial features and inherent biases in the clustering algorithm generate a large number of erroneous pseudo-labels. When these unreliable pseudo-labels are used as supervisory signals to train hash networks, they severely mislead the direction of model parameter updates, resulting in degraded model performance. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a semantically guided discriminative and robust unsupervised cross-modal hashing method. This hashing method trains a cross-modal hashing model with strong generalization and high robustness by constructing a discriminative semantic clustering learning mechanism and an adaptive robust prototype supervised learning mechanism.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following means:

[0008] A semantically guided, discriminative, and robust unsupervised cross-modal hashing method includes the following steps:

[0009] S1. Obtain the multimodal training dataset of text and images to obtain the preprocessed batch training dataset;

[0010] S2. Construct and initialize the multimodal feature extractor and the soft symbolic hash layer;

[0011] S3. Based on a multimodal feature extractor and a soft symbolic hashing layer, a cross-modal hashing model is trained using unsupervised data under the guidance of discriminative semantic clustering learning and adaptive robust prototype supervised learning mechanisms.

[0012] S4. Based on the obtained cross-modal hashing model, text or images are used as input to retrieve semantic matching results from the image or text pool.

[0013] Furthermore, step S1 is specifically divided into the following steps:

[0014] S101. Obtain a standard image dataset without labels;

[0015] S102. Obtain a text dataset that is unlabeled and associated with the description of the image dataset, and combine it with the image dataset to form an unsupervised training multimodal dataset;

[0016] S103. Perform data cleaning on the multimodal dataset to obtain a preprocessed multimodal dataset. ;

[0017] S104. Divide and organize the preprocessed multimodal data to obtain the batch training dataset X.

[0018] Furthermore, step S2 is specifically divided into the following steps:

[0019] S201. Establish and initialize the VisionTransformer network framework with pre-trained parameters to obtain the image feature extractor g. i ;

[0020] S202. Establish and initialize the BERT network framework with pre-trained parameters to obtain the text feature extractor g. t ;

[0021] S203. Build soft symbolic hash layers f after the two modal backbone network models respectively. i and f t ;

[0022] S204, Combined Image Feature Extractor g i Text feature extractor g t Image soft symbol hash layer f i and text soft symbol hash layer f t The cross-modal hash model structure is obtained. .

[0023] Furthermore, step S3 is specifically divided into the following steps:

[0024] S301, Based on the batch training image data X={X i X t}, which is then input into the multimodal feature extractor g i and g t In the process, the deep features x={x} of each batch of images and text are extracted. i ,x t};

[0025] S302. Input the deep features of the image and text into the corresponding soft symbolic hash layer f respectively. i and f t In the process, the deep hash codes corresponding to the image and text are obtained. ;

[0026] S303. After concatenating the deep hash codes of the two modalities, the K-Means clustering algorithm is used to obtain a unified semantic prototype for the modalities. ;

[0027] S304. Calculate the contrastive loss L based on semantic prototype. cd Furthermore, it uses contrastive loss to guide the optimization of cross-modal hashing models, realizes a discriminative semantic clustering learning mechanism, and guides the learning of cross-modal hashing models;

[0028] S305, Calculating Robust Generalized Cross-Entropy Loss L Based on Semantic Prototype rp Furthermore, cross-entropy loss is used to guide the optimization of the cross-modal hashing model, realizing an adaptive robust prototype supervised learning mechanism to guide the learning of the cross-modal hashing model;

[0029] S306. Based on the semantic prototype-based contrastive loss and robust generalized cross-entropy loss, the total loss L of the entire semantically guided unsupervised cross-modal hashing model optimization with both discriminative and robust capabilities is calculated.

[0030] S307. Calculate the gradient of the total loss, and perform backpropagation and parameter optimization on the entire model;

[0031] S308. Determine whether the loss meets the convergence condition. If yes, obtain the semantically guided unsupervised cross-modal hash model with discriminative power and robustness that has been trained and proceed to step S4. Otherwise, return to step S301.

[0032] Furthermore, the expression for the multimodal depth features in step S301 is as follows:

[0033]

[0034] in, Let g represent the depth features of the j-th image and the depth features of the text, respectively. i and g t X represents the feature extractor for the image and the feature extractor for the text, respectively. i and X t These represent multimodal batch training data for images and text, respectively.

[0035] Furthermore, the expression corresponding to the soft symbol hash layer in step S302 is as follows:

[0036]

[0037] in, Let f represent the depth hash code of the j-th image and the depth hash code of the text, respectively. i and f t These represent soft symbol hash encoders, Let represent the depth features of the j-th image and the depth features of the text, respectively, and let represent the numerical values ​​of the corresponding dimensions of the features input to the soft symbol hash encoder.

[0038] Furthermore, the expression for the semantic prototype center in step S303 is as follows:

[0039]

[0040] Among them, c i h represents the semantic prototype center. j S represents the concatenated cross-modal fusion hash code. i h represents the cluster obtained by K-Means clustering. j ∈S i Indicates if h j Belonging to the i-th sample cluster S i ,|S j | indicates the number of samples in the cluster, and Concat represents the feature concatenation function.

[0041] Furthermore, in step S304, the contrastive loss L based on semantic prototypes... cd The expression is as follows:

[0042]

[0043] Among them, L cd This represents a contrastive loss function based on semantic prototypes. This represents the number of samples in the training batch, and sim represents the cosine similarity calculation. h j The cluster prototype center to which it belongs, h j The set of prototype centers that do not belong to h j The k-th prototype center in the set of prototype centers that does not belong, where j and k represent index numbers. This indicates an adjustable temperature coefficient.

[0044] Furthermore, the expression for the robust generalized cross-entropy loss in step S305 is as follows:

[0045]

[0046] Among them, L rp Let K represent the robust generalized cross-entropy loss function, and K represent the number of semantic prototype center clusters. σ represents the sample reliability parameter, calculated from the sample prediction probability; σ represents the Sigmoid function; and α represents the hyperparameter, used to control the magnitude of change in the reliability parameter. Information entropy, p, represents the predicted probability of a sample. jk The probability of classification prediction guided by the semantic prototype center is expressed as follows:

[0047]

[0048] Among them, c j c l This represents the semantic prototype center, and j, k, and l represent index numbers.

[0049] Furthermore, the expression for the total loss in step S306 is:

[0050]

[0051] Where L represents the total loss function and λ represents the weight value.

[0052] Furthermore, the convergence condition in step S308 is as follows:

[0053] The total loss of 5 consecutive batches meets the requirements. , where L ep L represents the total loss of the ep-th batch. ep-1 Let |·| represent the total loss of the (ep-1)th batch, and |·| represent the absolute value function.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This invention leverages the native semantic information carried by multimodal samples to achieve multimodal feature extraction of images and text using a pre-trained encoder and attention mechanism. This endows the model with the ability to comprehensively extract fine-grained features and overall features of images and text. The acquired trained multimodal data is processed through a cross-modal hashing model composed of a pre-constructed feature extractor and a soft symbolic hashing layer to extract the fine-grained dimensional features and feature dimensional characteristics of images. These features are then mapped to Hamming space to obtain deep hash codes, representing a more efficient encoding while maintaining the original semantics of the data.

[0056] By using an adaptive robust prototype supervised learning mechanism and the guidance and supervision of the learning mechanism, the original semantics of multimodal data guide the hash codes to cluster into the same category, and the semantic center is used as the category center (pseudo-label) of the hash code for unsupervised classification. This avoids the problem of noisy pseudo-labels caused by the lack of labels, realizes the classification discriminative power of hash codes and robust hash representation learning under pseudo-labels, and achieves efficient cross-modal retrieval. Attached Figure Description

[0057] Figure 1 This is a flowchart of the unsupervised cross-modal hashing method of the present invention;

[0058] Figure 2 This is a flowchart illustrating the training process of the unsupervised cross-modal hashing model in this invention. Detailed Implementation

[0059] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples.

[0060] like Figure 1 As shown, a semantically guided, discriminative, and robust unsupervised cross-modal hashing method includes the following steps:

[0061] S1. Obtain the multimodal training dataset of text and images to obtain the preprocessed batch training dataset. The specific steps are as follows:

[0062] S101. Obtain a standard image dataset without labels;

[0063] S102. Obtain a text dataset that is unlabeled and associated with the description of the image dataset, and combine it with the image dataset to form an unsupervised training multimodal dataset;

[0064] S103. Perform data cleaning on the multimodal dataset to obtain a preprocessed multimodal dataset. ;

[0065] S104. Divide and organize the preprocessed multimodal data to obtain the batch training dataset X.

[0066] S2. Construct and initialize the multimodal feature extractor and soft symbolic hashing layer, with the following specific steps:

[0067] S201. Establish and initialize the VisionTransformer network framework with pre-trained parameters to obtain the image feature extractor g. i ;

[0068] S202. Establish and initialize the BERT network framework with pre-trained parameters to obtain the text feature extractor g. t ;

[0069] S203. Build soft symbolic hash layers f after the two modal backbone network models respectively. i and f t ;

[0070] S204, Combined Image Feature Extractor g i Text feature extractor g t Image soft symbol hash layer f i and text soft symbol hash layer f t The cross-modal hash model structure is obtained. .

[0071] S3. Based on a multimodal feature extractor and a soft symbolic hashing layer, a cross-modal hashing model is trained using unsupervised data under the guidance of discriminative semantic clustering learning and adaptive robust prototype supervised learning mechanisms. The specific steps are as follows:

[0072] S301, Based on the batch training image data X={X i X t}, which is then input into the multimodal feature extractor g i and g t In the process, the deep features x={x} of each batch of images and text are extracted. i ,x t The expression for multimodal deep features is as follows:

[0073]

[0074] in, Let g represent the depth features of the j-th image and the depth features of the text, respectively. i and g tX represents the feature extractor for the image and the feature extractor for the text, respectively. i and X t These represent multimodal batch training data for images and text, respectively.

[0075] S302. Input the deep features of the image and text into the corresponding soft symbolic hash layer f respectively. i and f t In the process, the deep hash codes corresponding to the image and text are obtained. The expression corresponding to the soft symbol hash layer is as follows:

[0076]

[0077] in, Let f represent the depth hash code of the j-th image and the depth hash code of the text, respectively. i and f t These represent soft symbol hash encoders, Let represent the depth features of the j-th image and the depth features of the text, respectively, and let represent the numerical values ​​of the corresponding dimensions of the features input to the soft symbol hash encoder.

[0078] S303. After concatenating the deep hash codes of the two modalities, the K-Means clustering algorithm is used to obtain a unified semantic prototype for the modalities. The expression for the semantic prototype center is as follows:

[0079]

[0080] Among them, c i h represents the semantic prototype center. j S represents the concatenated cross-modal fusion hash code. i h represents the cluster obtained by K-Means clustering. j ∈S i Indicates if h j Belonging to the i-th sample cluster S i ,|S j | indicates the number of samples in the cluster, and Concat represents the feature concatenation function.

[0081] S304. Calculate the contrastive loss L based on semantic prototype. cd Furthermore, it uses contrastive loss to guide the optimization of cross-modal hashing models, realizing a discriminative semantic clustering learning mechanism to guide the learning of cross-modal hashing models, based on the contrastive loss L... cd The expression is as follows:

[0082]

[0083] Among them, L cdThis represents a contrastive loss function based on semantic prototypes. This represents the number of samples in the training batch, and sim represents the cosine similarity calculation. h j The cluster prototype center to which it belongs, h j The set of prototype centers that do not belong to h j The k-th prototype center in the set of prototype centers that does not belong, where j and k represent index numbers. This indicates an adjustable temperature coefficient.

[0084] The semantic prototype-based contrastive loss function encourages samples to align more closely with their corresponding semantic prototypes, thereby maintaining intra-cluster semantic consistency. Furthermore, it evenly pushes samples away from unrelated clusters, effectively accommodating the boundaries between semantically different clusters, thus achieving inter-cluster separability and ultimately realizing a discriminative hash code representation in Hamming space.

[0085] S305, Calculating Robust Generalized Cross-Entropy Loss L Based on Semantic Prototype rp Furthermore, cross-entropy loss is used to guide the optimization of the cross-modal hashing model, realizing an adaptive robust prototype supervised learning mechanism to guide the learning of the cross-modal hashing model; the expression for robust generalized cross-entropy loss is as follows:

[0086]

[0087] Among them, L rp Let K represent the robust generalized cross-entropy loss function, and K represent the number of semantic prototype center clusters. σ represents the sample reliability parameter, calculated from the sample prediction probability; σ represents the Sigmoid function; and α represents the hyperparameter, used to control the magnitude of change in the reliability parameter. Information entropy, p, represents the predicted probability of a sample. jk The probability of classification prediction guided by the semantic prototype center is expressed as follows:

[0088]

[0089] Among them, c j c l This represents the semantic prototype center, and j, k, and l represent index numbers.

[0090] When the prediction distribution exhibits low entropy, corresponding to higher model confidence, the parameter q tends to a smaller value. In this case, the loss function approximates the standard cross-entropy loss, thus promoting discriminative learning. When the prediction distribution has high entropy, it indicates uncertainty in the model's predictions, and the parameter q adaptively increases. Therefore, the loss function shifts towards noise-robust mean absolute error behavior, mitigating the impact of unreliable supervision signals and improving the model's robustness by using the prototype center as a guide.

[0091] S306. Based on the semantic prototype-based contrastive loss and robust generalized cross-entropy loss, the total loss L of the semantically guided unsupervised cross-modal hashing model optimization, which combines discriminative power and robustness, is calculated; the expression for the total loss is:

[0092]

[0093] Where L represents the total loss function and λ represents the weight value.

[0094] By combining a weighted fusion approach with a semantic prototype-based contrastive loss function and a robust generalized cross-entropy loss function, robust pseudo-label supervision guided by the semantic prototype center is achieved during model training. This promotes the model to achieve a representation that can balance the discriminative power and robustness of multimodal hash codes, ultimately enabling cross-modal hash retrieval.

[0095] S307. Calculate the gradient of the total loss, and perform backpropagation and parameter optimization on the entire model;

[0096] S308. Determine if the loss satisfies the convergence condition. If yes, obtain the semantically guided, discriminative, and robust unsupervised cross-modal hashing model that has been trained and proceed to step S4. Otherwise, return to step S301. The convergence condition is as follows:

[0097] The total loss of 5 consecutive batches meets the requirements. , where L ep L represents the total loss of the ep-th batch. ep-1 Let |·| represent the total loss of the (ep-1)th batch, and |·| represent the absolute value function.

[0098] S4. Based on the obtained cross-modal hashing model, text or images are used as input to retrieve semantic matching results from the image or text pool.

[0099] The above description is merely an embodiment of the present invention. It should be reiterated that those skilled in the art can make several improvements to the present invention without departing from the principle of the present invention, and these improvements are also included within the scope of protection of the claims of the present invention.

Claims

1. A semantically guided, discriminative, and robust unsupervised cross-modal hashing method, characterized in that: Includes the following steps: S1. Obtain the multimodal training dataset of text and images to obtain the preprocessed batch training dataset; S2. Construct and initialize the multimodal feature extractor and the soft symbolic hash layer; S3. Based on a multimodal feature extractor and a soft symbolic hashing layer, a cross-modal hashing model is trained using unsupervised data under the guidance of discriminative semantic clustering learning and adaptive robust prototype supervised learning mechanisms. S4. Based on the obtained cross-modal hashing model, text or images are used as input to retrieve semantic matching results from the image or text pool.

2. The semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 1, characterized in that: The specific steps of step S1 are as follows: S101. Obtain a standard image dataset without labels; S102. Obtain a text dataset that is unlabeled and associated with the description of the image dataset, and combine it with the image dataset to form an unsupervised training multimodal dataset; S103. Perform data cleaning on the multimodal dataset to obtain a preprocessed multimodal dataset. ; S104. Divide and organize the preprocessed multimodal data to obtain the batch training dataset X.

3. The semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 1, characterized in that: The specific steps of step S2 are as follows: S201. Establish and initialize the VisionTransformer network framework with pre-trained parameters to obtain the image feature extractor g. i ; S202. Establish and initialize the BERT network framework with pre-trained parameters to obtain the text feature extractor g. t ; S203. Build soft symbolic hash layers f after the two modal backbone network models respectively. i and f t ; S204, Combined Image Feature Extractor g i Text feature extractor g t Image soft symbol hash layer f i and text soft symbol hash layer f t The cross-modal hash model structure is obtained. .

4. The semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 1, characterized in that: The specific steps of step S3 are as follows: S301, Based on the batch training image data X={X i X t }, which is then input into the multimodal feature extractor g i and g t In the process, the deep features x={x} of each batch of images and text are extracted. i ,x t }; S302. Input the deep features of the image and text into the corresponding soft symbolic hash layer f respectively. i and f t In the process, the deep hash codes corresponding to the image and text are obtained. ; S303. After concatenating the deep hash codes of the two modalities, the K-Means clustering algorithm is used to obtain a unified semantic prototype for the modalities. ; S304. Calculate the contrastive loss L based on semantic prototype. cd Furthermore, it uses contrastive loss to guide the optimization of cross-modal hashing models, realizes a discriminative semantic clustering learning mechanism, and guides the learning of cross-modal hashing models; S305, Calculating Robust Generalized Cross-Entropy Loss L Based on Semantic Prototype rp Furthermore, cross-entropy loss is used to guide the optimization of the cross-modal hashing model, realizing an adaptive robust prototype supervised learning mechanism to guide the learning of the cross-modal hashing model; S306. Based on the semantic prototype-based contrastive loss and robust generalized cross-entropy loss, the total loss L of the semantically guided unsupervised cross-modal hashing model optimization, which combines discriminative power and robustness, is calculated. The expression is: Where L represents the total loss function and λ represents the weight value; S307. Calculate the gradient of the total loss, and perform backpropagation and parameter optimization on the entire model; S308. Determine whether the loss meets the convergence condition. If yes, obtain the semantically guided unsupervised cross-modal hash model with discriminative power and robustness that has been trained and proceed to step S4. Otherwise, return to step S301.

5. The semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 4, characterized in that: The expression for the multimodal depth feature in step S301 is as follows: in, Let g represent the depth features of the j-th image and the depth features of the text, respectively. i and g t X represents the feature extractor for the image and the feature extractor for the text, respectively. i and X t These represent multimodal batch training data for images and text, respectively.

6. The semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 5, characterized in that: The expression corresponding to the soft symbol hash layer in step S302 is as follows: in, Let f represent the depth hash code of the j-th image and the depth hash code of the text, respectively. i and f t These represent soft symbol hash encoders, Let represent the depth features of the j-th image and the depth features of the text, respectively, and let represent the numerical values ​​of the corresponding dimensions of the features input to the soft symbol hash encoder.

7. A semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 6, characterized in that: The expression for the semantic prototype center in step S303 is as follows: Among them, c i h represents the semantic prototype center. j S represents the concatenated cross-modal fusion hash code. i h represents the cluster obtained by K-Means clustering. j ∈S i Indicates if h j Belonging to the i-th sample cluster S i ,|S j | indicates the number of samples in the cluster, and Concat represents the feature concatenation function.

8. The semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 7, characterized in that: In step S304, the contrastive loss based on semantic prototype L cd The expression is as follows: Among them, L cd This represents a contrastive loss function based on semantic prototypes. This represents the number of samples in the training batch, and sim represents the cosine similarity calculation. h j The cluster prototype center to which it belongs, h j The set of prototype centers that do not belong to h j The k-th prototype center in the set of prototype centers that does not belong, where j and k represent index numbers. This indicates an adjustable temperature coefficient.

9. A semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 7, characterized in that: The expression for the robust generalized cross-entropy loss in step S305 is as follows: Among them, L rp Let K represent the robust generalized cross-entropy loss function, and K represent the number of semantic prototype center clusters. σ represents the sample reliability parameter, calculated from the sample prediction probability; σ represents the Sigmoid function; and α represents the hyperparameter, used to control the magnitude of change in the reliability parameter. The information entropy, p, represents the predicted probability of a sample. jk The probability of classification prediction guided by the semantic prototype center is expressed as follows: Among them, c j c l This represents the semantic prototype center, and j, k, and l represent index numbers.

10. A semantically guided, discriminative, and robust unsupervised cross-modal hashing method according to claim 4, characterized in that: The convergence condition in step S308 is as follows: The total loss of 5 consecutive batches meets the requirements. , where L ep L represents the total loss of the ep-th batch. ep-1 Let |·| represent the total loss of the (ep-1)th batch, and |·| represent the absolute value function.

Images