A Label-Enhanced Supervised Multimodal Hash Retrieval Method and System

Abstract

This invention discloses a supervised multimodal hash retrieval method and system based on label enhancement, belonging to the field of artificial intelligence and multimedia retrieval technology. The technical problem this invention aims to solve is how to better capture the similarity information between multimodal data points and achieve better performance and accuracy in multimodal retrieval tasks. The technical solution includes: data preprocessing: acquiring and organizing public datasets of image and text modalities, and dividing each public dataset into training, testing, and retrieval datasets; extracting deep features: using a pre-trained network model to extract features from the raw data of the public datasets of image and text modalities respectively, obtaining deep features of the image modality and the text modality; offline training; variable update and optimization; and online query. The system includes a data preprocessing unit, a feature extraction unit, an offline training unit, a variable update and optimization unit, and an online query unit.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimedia retrieval technology, specifically a supervised multimodal hash retrieval method and system based on tag enhancement. Background Technology

[0002] With revolutionary advancements in communication technology and the widespread adoption of the internet, people can easily upload, download, and share multimedia data, including images, audio, and video, through diverse online platforms and applications. However, in practical applications, as the scale and volume of multimedia data continue to increase, users' demand for similarity search technologies capable of handling multiple modalities is becoming increasingly urgent. Multimodal hashing technology provides a cutting-edge and effective solution for processing large-scale multimedia data. This technology can transform high-dimensional multimodal features into compact binary hash codes, achieving efficient data storage. Simultaneously, through XOR operations between hash codes, similarity retrieval can be achieved quickly and accurately, greatly improving retrieval efficiency. Multimodal hashing technology has broad application prospects, especially in the field of multimedia data processing, providing a new approach to solving the similarity search problem for large-scale data.

[0003] Existing multimodal hashing methods mainly focus on two learning approaches: unsupervised and supervised. Unsupervised multimodal hashing methods primarily utilize the similarity between data points to learn the hash function. While unsupervised multimodal hashing methods offer greater flexibility in processing unlabeled data, they may not fully leverage existing label information to improve retrieval accuracy. Supervised multimodal hashing methods, on the other hand, utilize existing label information to enhance the discriminative power of the hash code, thereby more accurately reflecting the underlying semantics of multimodal data and improving the accuracy of similarity searches.

[0004] While supervised multimodal hashing methods can more accurately reflect the underlying semantics of multimodal data and improve the accuracy of similarity searches, existing supervised hashing methods generally tend to oversimplify the use of label information. This simplification is mainly reflected in the use of discrete binary labels to determine whether a data point belongs to a specific category. However, this still has limitations in effectively capturing the true similarity relationships between multimodal data points, mainly in the following aspects:

[0005] First, discrete binary labels are too simplistic and direct, focusing only on predicting whether a data point belongs to a certain category, without capturing information about the degree of similarity or dissimilarity between different samples within the same category. For example, suppose there is an animal image dataset containing three images of cats. Each image is assigned a binary label based on the type of animal it depicts. Using discrete binary labels would result in assigning the same label to all three cat images, indicating that they belong to the "cat" category. However, this approach fails to capture any information about the similarity or dissimilarity between images within the "cat" category.

[0006] Secondly, discrete binary labeling assumes that samples labeled 0 do not reflect similarity or correlation within a class. However, in real-world scenarios, these samples may exhibit varying degrees of similarity or correlation. For example, in animal images, two images of different species may still share some common visual features or characteristics, making them similar to some extent. Discrete binary labeling fails to capture these subtle differences and treats all samples labeled 0 as completely different or unrelated, thus losing crucial similarity information.

[0007] Furthermore, the discrete constraints imposed on hash codes limit their expressive power, failing to fully capture the subtle semantic information contained in the labels. This limitation stems from the loss of details and nuances during binary quantization, as well as the lack of inherent continuity in binary representation.

[0008] These limitations can lead to a decrease in performance and accuracy in multimodal hash retrieval tasks.

[0009] Therefore, how to better capture the similarity information between multimodal data points and achieve better performance and accuracy in multimodal retrieval tasks is a technical problem that urgently needs to be solved. Summary of the Invention

[0010] The technical objective of this invention is to provide a label-enhanced supervised multimodal hash retrieval method and system to address the problem of how to better capture similarity information between multimodal data points and achieve better performance and accuracy in multimodal retrieval tasks.

[0011] The technical objective of this invention is achieved as follows: a label-enhanced supervised multimodal hash retrieval method, the specific method of which is as follows:

[0012] Data preprocessing: Obtain and organize public datasets of image and text modalities, and divide each public dataset into training dataset, test dataset, and retrieval dataset for subsequent use;

[0013] Extracting deep features from data: By using pre-trained network models to extract features from the raw data of public datasets for both image and text modalities, we can obtain deep features for both image and text modalities, which can more accurately reflect the underlying semantics of the data.

[0014] Offline training: The objective function is trained using image modal depth features and text modal depth features and their corresponding labels. The continuous and dynamic adjustment of discrete labels and the representation learning of optimized hash codes are also considered to learn a reliable objective function.

[0015] Variable update optimization: The alternating optimization method is used to train and optimize each variable in the objective function, thereby achieving the goal of comprehensively updating and optimizing the objective function;

[0016] Online query: A hash function is constructed using hash codes generated offline. The hash function is used to generate hash codes for new query samples. Similarity retrieval is then performed by calculating Hamming distance. The average retrieval performance is calculated through multiple iterations, and the mAP value is obtained to evaluate the retrieval performance.

[0017] As a preferred method, the data preprocessing is as follows:

[0018] We obtained and downloaded public datasets containing image and text modalities from public network resources as the original datasets; these public datasets included the MIR Flickr dataset, the NUS-WIDE dataset, and the MS COCO dataset.

[0019] The downloaded raw dataset is partitioned as follows:

[0020] Randomly select a subset of image-text pairs from the MIR Flickr dataset, NUS-WIDE dataset, and MS COCO dataset as the test set to evaluate test performance;

[0021] Excluding samples from the test set, the remaining image-text pairs in the MIR Flickr dataset, NUS-WIDE dataset, and MS COCO dataset are used as the retrieval dataset for retrieval operations;

[0022] In the retrieval dataset, any portion of image-text pairs is randomly selected as the training set for training the objective function, in order to achieve better system retrieval performance.

[0023] More specifically, the extraction of deep features from the data is as follows:

[0024] Image modal depth feature extraction: A ViT model pre-trained on the large-scale ImageNet dataset is used as the image encoder to extract features from the image; specifically as follows:

[0025] The input image is uniformly resized to H×W pixels to obtain the reshaped image;

[0026] Convert the reshaped image into flattened two-dimensional planar image patches;

[0027] Introducing position embedding to capture spatial relationships between image patches;

[0028] Feature extraction is performed by stacking multiple Transformer layers. Each Transformer layer includes a self-attention mechanism and a fully connected feedforward network. Global average pooling is performed on the output of all positions of the last Transformer layer to obtain the global average pooling representation of the entire image.

[0029] Linear projection is performed on the global average pooling representation of the entire image to obtain the image's embedding feature representation, which is a vector containing 4096-dimensional features. This 4096-dimensional feature vector can capture the high-level semantic information of the image.

[0030] Extracting deep features from text modalities: Using the BERT model as a text encoder, features are extracted from the text modalities; specifically as follows:

[0031] The input text is divided into tokens, and each token is converted into a vector representation through an embedding layer;

[0032] Introducing positional encoding, which considers the positional information of words in a sentence;

[0033] By using a multi-head self-attention mechanism network, different parts are focused on at different positions in the input sequence to capture complex contextual relationships. A feedforward neural network is applied after each attention layer to perform nonlinear transformation on the features.

[0034] After multi-head self-attention and feedforward neural network operations, layer normalization is applied to obtain the deep feature representation of the text. Among them, the feature extraction dimension values ​​of the text data are different on three different datasets: MIR Flickr dataset, NUS-WIDE dataset and MS COCO dataset.

[0035] Better yet, offline training is specifically as follows:

[0036] To optimize the effectiveness of similarity supervision, an objective function is constructed that aligns the hash code with the label matrix, as shown in the formula:

[0037] stCC T =I c×c B∈{-1,1};

[0038] Where B represents the binary hash code, B = [b1; b2; ...; b N ]∈{-1,1} N×k N represents the number of training data samples; k represents the length of the hash code; L represents the discrete label matrix, L∈{0,1} N×c ;c represents the total number of categories covered by all data;I c×c It is the identity matrix; the transformation matrix. Subject to CC T =I c×c The constraints ensure the orthogonality of matrix C;

[0039] By introducing label information through a label matrix L, the embedded label information effectively maintains inter-modal and intra-modal similarity, achieving better information retrieval performance. Aligning the hash code with the label matrix can further enhance the preservation of semantic similarity. The aligned semantic information serves as guidance during training, helping to improve the overall performance of multimodal hashing.

[0040] Constructing a tag enhancement strategy: By defining the enhanced tag matrix This enhances label relevance; where L is a pre-labeled discrete multi-label matrix, L∈{0,1}. N×c L is associated with one or more labels, providing additional semantic information for multimodal data; in the label matrix L, element l ij The matrix M indicates whether the i-th sample belongs to the j-th class (1 for belonging, 0 for not belonging); it is an indicator matrix that records the relationship between each sample and each class, determining whether a sample belongs to a certain class; if a sample belongs to any class, the corresponding element is 1, otherwise it is 0; the matrix E is an adjustment matrix used to adjust the mapping process from discrete labels to continuous labels. The goal of matrix E is to measure the degree of adjustment required for each element in the original binary label matrix to align with the continuous label representation. This adjustment mapping aligns the discrete labels to the continuous label representation, specifically as follows: the slack variable ε is dynamically adjusted according to the class membership of each class; and by converting the original discrete binary labels into continuous and adjustable labels, the similarity between different classes can be captured more accurately, thus better capturing the intrinsic differences between classes;

[0041] Augmented label hashing learning: using an augmented label matrix to capture the subtle differences between different categories. The original label matrix L is replaced as input to better distinguish different categories and perform more accurate retrieval. The objective function for enhancing the label hash learning process is as follows:

[0042]

[0043] CC T =I c×c ,B{-1,1};

[0044] Where α and γ are non-negative balancing parameters; c represents the total number of categories covered by all data. Enhanced tag matrix Regularization learning is used to prevent overfitting and improve the model's generalization ability.

[0045] Fusion of Heterogeneous Multimodal Features: To effectively address the heterogeneity gap problem, heterogeneous multimodal features are encoded into a shared space, allowing direct comparison of data from different modalities. Specifically, dynamic modality weighting parameters are used to fully consider the complementarity of multimodal features, thereby improving the expressive and discriminative power of the learned hash codes. The objective function of the heterogeneous multimodal feature fusion process is as follows:

[0046]

[0047] in, It is a nonlinear embedding representation of the v-th modality, which captures the features of each modality by utilizing the relationship between randomly selected m anchor points and data samples. Matrix U v It is the orthogonal rotation matrix of the v-th mode. By mapping the nonlinear embedding of each mode to a common hash code space, the hash code of the specific features of each mode is effectively learned and preserved. To effectively capture the unique characteristics and advantages of each mode while taking into account the complementarity between modes, dynamic modal weighting parameters are introduced. This allows for flexible adjustment of the importance and weight contribution of each modality;

[0048] Construction of the final objective function: The final objective function is obtained by integrating the objective function of the enhanced label hashing learning process and the objective function of the heterogeneous multimodal feature fusion process, as follows:

[0049]

[0050] stU v (U v ) T =I m×m CC T =I c×c B∈{-1,1};

[0051] in, Hash codes are learned by minimizing the reconstruction error between the nonlinear embedding and the learned hash, focusing on the nonlinear representation capability of the learned hash codes for the original data; The discrete labels are converted into continuous and adjustable labels. The purpose of this is to effectively distinguish the differences between various categories and provide more detailed semantic relationship information. By adjusting the elements in the label matrix, each sample can be more finely assigned to the corresponding category. Used to prevent overfitting and improve the system's generalization ability.

[0052] More preferably, variable update optimization obtains the enhanced label matrix for supervised hash code learning through an alternating optimization method. The orthogonal rotation matrix U used to map the nonlinear embedding of the v-th mode to a common hash code space v The binary hash code B is used for efficient retrieval, and the transformation matrix C is used to enhance semantic similarity preservation and fuse complementary features between different modalities; specifically as follows:

[0053] Optimization and Update Specifically as follows:

[0054] In removal and After removing irrelevant terms, the objective function simplifies to

[0055] Will Transforming the matrix into trace operation form helps to handle matrix operations more efficiently, reduce computational complexity, and improve computational efficiency; further... Find the partial derivative and then... Setting it to zero, we obtain the update function for the discrete label matrix.

[0056] By expanding the discrete label matrix The scope is expanded to further enhance the effectiveness of similarity supervision; and the discrete label matrix is ​​extended to... Specifically, the update result for the error category is restricted to be less than 0, while the update result for the true category should be greater than 1; thus adjusting... The specific optimization and update formula is as follows:

[0057]

[0058] Optimize and update U v The details are as follows:

[0059] Treating other variables as constants, the objective function is transformed into...

[0060] Among them, in U v (U v ) T =I m×m Under constraints, U v The optimal solution is transformed into solving The problem of maximizing the trace; the goal is to maximize Equivalent to maximizing Tr((U) v ) T G); U is derived using the Singular Value Decomposition (SVD) theorem. v The optimal solution is Among them, G l and G r These are the left and right singular vectors of G, respectively;

[0061] Optimize and update C as follows:

[0062] With other parameters fixed, the objective function for updating the transformation matrix C is expressed as:

[0063] Among them, in CC T =I c×c Under constraints, the objective function is transformed using the singular value decomposition principle. Transform it into a maximization problem, and use the singular value decomposition theorem to derive the optimal solution for C. Solve for the left and right singular vectors;

[0064] Optimize and update B as follows:

[0065] Keeping all other variables constant, the objective function associated with variable B is:

[0066] Taking the partial derivative with respect to B and setting B to zero, we derive a closed-form solution for B as follows:

[0067] More preferably, the online query utilizes the hash codes generated during the offline training phase to construct a hash function, which is used to generate hash codes for new query samples and obtain a linear projection matrix W for multimodal feature learning. Then, the symbolic function sgn() is used to obtain the binary hash code representations of the test dataset and the retrieval dataset samples respectively. The Hamming distance between the hash codes of the test dataset and the retrieval dataset samples is calculated. The retrieval dataset samples are sorted in ascending order according to the corresponding Hamming distances to obtain the final retrieval results. Finally, the mAP value is used to evaluate the retrieval performance. Specifically:

[0068] Learning the hash code of out-of-sample query data: The hash function learning process is transformed into a classification problem. By concatenating the nonlinear embeddings of multimodal features, the projection matrix W is learned to represent the multimodal features. The specific objective function is as follows:

[0069]

[0070] in, Nonlinear embeddings representing multimodal features W is the projection matrix for multimodal feature learning. The objective function simplifies the training process and improves system efficiency and scalability; the formula for solving W is:

[0071]

[0072] Generate hash codes for query samples: Multimodal retrieval is performed using hash codes generated by a hash function. The specific formula for predicting the hash code for the query sample is as follows:

[0073]

[0074] Among them, X q This is a sample of query data. B q It is X q Hash code generated by a hash function; This represents the non-linear embedding of the query sample. n q This indicates the number of query samples; the linear projection matrix W is obtained during the training phase.

[0075] Calculate Hamming distance: Select any test sample in the test dataset and obtain the Hamming distance between the hash code of the test sample and the hash codes of all samples in the retrieval dataset; sort the samples in the retrieval dataset according to their Hamming distances and store the relationship between the test sample distances and a set threshold.

[0076] If the distance between the test samples is less than the set threshold, they will be ranked higher.

[0077] If the distance to the test sample is greater than the set threshold, it will be ranked lower, thus verifying the accuracy of the retrieval.

[0078] For each modality, the average precision (AP) at different recall rates is calculated, and the average precision of each modality is averaged to obtain the mAP value of each modality.

[0079] A label-enhanced supervised multimodal hash retrieval system, comprising:

[0080] The data preprocessing unit is used to acquire and organize public datasets of image and text modalities, and divide each public dataset into training dataset, test dataset and retrieval dataset for subsequent use;

[0081] The feature extraction unit is used to extract features from the raw data of the public datasets of image modality and text modality respectively through a pre-trained network model, and obtain the deep features of image modality and text modality, which can more accurately reflect the underlying semantics of the data;

[0082] The offline training unit is used to train the objective function using image modal depth features and text modal depth features and their corresponding labels, and comprehensively considers the continuous and dynamic adjustment of discrete labels and the representation learning of optimized hash codes, thereby learning a reliable objective function.

[0083] The variable update and optimization unit is used to train and optimize each variable in the objective function using an alternating optimization method, thereby achieving the goal of comprehensively updating and optimizing the objective function;

[0084] The online query unit constructs a hash function using hash codes generated offline during training. This hash function generates hash codes for new query samples, and similarity retrieval is performed by calculating Hamming distance. The average retrieval performance is calculated through multiple iterations, and the mAP value is obtained to evaluate the retrieval performance. The mAP value is an average accuracy metric that measures the performance of tasks such as information retrieval or object detection.

[0085] Preferably, the feature extraction unit includes:

[0086] The image modality feature extraction module is used to extract features from images using a ViT model pre-trained on the ImageNet dataset as an image encoder;

[0087] The text modality feature extraction module is used to extract features from images using the BERT model as a text encoder;

[0088] The offline training unit includes:

[0089] The similarity supervision optimization module is used to optimize the effectiveness of similarity supervision through the objective function. stCC T =I c×c Align the hash code with the label matrix if B∈{-1,1};

[0090] The tag enhancement strategy building module is used to define the enhanced tag matrix. To enhance the relevance of tags;

[0091] An enhanced label hashing learning module is used to enhance the label matrix by capturing small differences between different categories. Instead of the original label matrix L, it is used as input to better distinguish different categories and perform more accurate retrieval;

[0092] The heterogeneous multimodal feature fusion module is used to encode heterogeneous multimodal features into a shared space, allowing direct comparison of data from different modalities within the shared space;

[0093] The final objective function construction module is used to obtain the final objective function by integrating the objective function of the enhanced label hashing learning process and the objective function of the fusion heterogeneous multimodal feature process;

[0094] The variable update optimization unit includes:

[0095] Optimize and update module one, used to remove and After removing irrelevant terms, simplify the objective function, and then transform the simplified objective function into trace form before performing further operations. Find the partial derivative and then... Setting it to zero yields the optimized update function for the discrete label matrix.

[0096] The second optimization module is used to transform the objective function and quantitatively optimize and update U using singular value decomposition, while treating other variables as constants. v ;

[0097] The third optimization module is used to transform the objective function into a maximization problem when other variables are considered constant, and to quantitatively optimize and update C using singular value decomposition.

[0098] The fourth optimization and update module is used to obtain the closed-form solution of B by taking the partial derivative of the objective function associated with variable B with respect to B, and setting B to zero, while treating other variables as constants, thereby completing the optimization and update of B.

[0099] An electronic device includes: a memory and at least one processor;

[0100] The memory contains computer programs;

[0101] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the label-enhanced supervised multimodal hash retrieval method as described above.

[0102] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the tag-enhanced supervised multimodal hash retrieval method described above.

[0103] The label-enhanced supervised multimodal hash retrieval method and system of the present invention have the following advantages:

[0104] (i) This invention improves the accuracy and efficiency of multimodal data retrieval, providing a superior solution for practical applications in fields such as image and text retrieval;

[0105] (ii) This invention employs pre-trained deep learning models ViT and BERT to extract deep feature representations of image and text modalities, respectively. This not only extracts rich information from multimodal data but also uses this information as input features for subsequent modules. At the same time, through deep learning technology, it can more accurately capture the underlying semantic information of images and text, thereby improving the accuracy and efficiency of retrieval.

[0106] (III) This invention proposes an effective label enhancement strategy that transforms discrete labels into continuous label values, effectively expanding the inter-class boundaries of discrete labels and further enriching the supervision signal of semantic similarity, thereby solving the problem that existing supervised multimodal hashing methods ignore label correlation information between different modalities. This strategy better reflects the complex relationships and semantic similarity between multimodal data, enabling more accurate classification and retrieval of multimodal data. In addition, this strategy can also improve the generalization ability of the model, enabling it to better adapt to constantly changing application scenarios.

[0107] (iv) This invention proposes an efficient hash code learning strategy. By dynamically adjusting image and text modal data, a multimodal fusion hash code is learned, which can better preserve the semantic information in the enhanced label information, while retaining the specific features of each modality, thereby enhancing the discriminative ability of the generated hash code.

[0108] (V) This invention proposes an effective alternating optimization strategy. During the offline training phase, by updating and optimizing the relevant objective function, it ensures that the variables involved are in the optimal configuration. This strategy synergistically optimizes the hash code and the enhanced tag matrix, aiming to better capture potential semantic relationships and improve overall retrieval performance.

[0109] (vi) This invention introduces a novel and unified multimodal hash learning framework that directly learns binary hash codes, effectively reducing the quantization error caused by relaxation strategies in existing methods. Furthermore, hash codes can be learned quickly, are simple to operate, and have high computational and storage efficiency—advantages required for large-scale multimedia retrieval. Attached Figure Description

[0110] The invention will be further described below with reference to the accompanying drawings.

[0111] Appendix Figure 1 The flowchart shows the label-enhanced supervised multimodal hash retrieval method.

[0112] Appendix Figure 2 A flowchart for data preprocessing;

[0113] Appendix Figure 3 A flowchart for extracting deep features from data;

[0114] Appendix Figure 4 This is a flowchart of the offline training process;

[0115] Appendix Figure 5 This is a flowchart of the offline training process;

[0116] Appendix Figure 6 A flowchart for online query process;

[0117] Appendix Figure 7 This is a schematic diagram of the structure of a label-enhanced supervised multimodal hash retrieval system.

[0118] Appendix Figure 8 This is a schematic diagram illustrating the working process of a label-enhanced supervised multimodal hash retrieval system.

[0119] Appendix Figure 9 A flowchart for extracting image modal depth features;

[0120] Appendix Figure 10 A flowchart for extracting deep features of text modalities. Detailed Implementation

[0121] The label-enhanced supervised multimodal hash retrieval method and system of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0122] Example 1:

[0123] As attached Figure 1 As shown in the figure, this embodiment provides a label-enhanced supervised multimodal hash retrieval method, which is as follows:

[0124] S1. Data Preprocessing: Obtain and organize the common datasets of image modalities and text modalities, and divide each common dataset into training dataset, test dataset and retrieval dataset for subsequent use;

[0125] S2. Extracting deep features from data: By using a pre-trained network model, features are extracted from the raw data of the public datasets for both image and text modalities to obtain deep features for both image and text modalities, which can more accurately reflect the underlying semantics of the data.

[0126] S3. Offline training: The objective function is trained using image modal depth features and text modal depth features and their corresponding labels. The continuous and dynamic adjustment of discrete labels and the representation learning of optimized hash codes are also considered to learn a reliable objective function.

[0127] S4. Variable Update Optimization: The alternating optimization method is used to train and optimize each variable in the objective function, thereby achieving the goal of comprehensively updating and optimizing the objective function;

[0128] S5. Online Query: Construct a hash function using the hash code generated by offline training, generate a hash code for new query samples using the hash function, perform similarity retrieval by calculating Hamming distance, calculate the average retrieval performance through multiple iterations, and obtain the mAP value to evaluate the retrieval performance.

[0129] As attached Figure 2 As shown, the data preprocessing in step S1 of this embodiment is as follows:

[0130] S101. Obtain and download public datasets containing image and text modalities from public network resources as the original datasets; among them, public datasets include the MIR Flickr dataset, the NUS-WIDE dataset, and the MS COCO dataset;

[0131] S102. Divide the downloaded raw dataset into data segments as follows:

[0132] S10201. Randomly select any portion of image-text pairs from the MIR Flickr dataset, NUS-WIDE dataset, and MS COCO dataset as the test set to evaluate test performance.

[0133] S10202. Exclude samples from the test set and use the remaining image-text pairs in the MIR Flickr dataset, NUS-WIDE dataset, and MSCOCO dataset as the retrieval dataset for retrieval operations;

[0134] S10203. In the retrieval dataset, randomly select any part of the image-text pairs as the training set for training the objective function to achieve better system retrieval performance.

[0135] As attached Figure 3 As shown, the extraction of data depth features in step S2 of this embodiment is as follows:

[0136] S201. Extract image modal depth features: as shown in the attached document. Figure 9 As shown, a ViT model pre-trained on the large-scale ImageNet dataset is used as an image encoder to extract features from the images; specifically as follows:

[0137] S20101. Preprocessing raw image modal data: The input image is uniformly adjusted to H×W pixel size to obtain the reshaped image; then the reshaped image is converted into flat two-dimensional planar image patches, and position embedding is introduced to capture the spatial relationship between the image patches;

[0138] S20102, Pre-trained ViT model: Feature extraction is performed by stacking multiple Transformer layers. Each Transformer layer includes a self-attention mechanism and a fully connected feedforward network. Global average pooling is performed on the output of all positions of the last Transformer layer to obtain the global average pooling representation of the entire image.

[0139] S20103. Extracting a 4096-dimensional feature vector: Perform a linear projection on the global average pooling representation of the entire image to obtain the image's embedded feature representation. The image's embedded feature representation is a vector containing 4096-dimensional features. This 4096-dimensional feature vector can capture the high-level semantic information of the image.

[0140] S202. Extract deep features of text modality: as shown in the appendix. Figure 10 As shown, the BERT model is used as a text encoder to extract features from the text modality; the details are as follows:

[0141] S20201. Preprocessing raw text modal data: Divide the input text into tokens, and convert each token into a vector representation through an embedding layer; and introduce positional encoding to consider the positional information of words in the sentence;

[0142] S20202, Pre-trained BERT model: Through a multi-head self-attention mechanism network, different parts are focused on at different positions in the input sequence to capture complex contextual relationships, and a feedforward neural network is applied after each attention layer to perform non-linear transformation on the features;

[0143] S20203, Extracting 1386-dimensional feature vectors from labeled data: After multi-head self-attention operation and feedforward neural network operation, layer normalization is applied to obtain the deep feature representation of the text; among them, the feature extraction dimension values ​​of the text data are different on three different datasets: MIR Flickr dataset, NUS-WIDE dataset and MS COCO dataset.

[0144] As attached Figure 4 As shown, the offline training in step S3 of this embodiment is as follows:

[0145] S301. Optimize the effectiveness of similarity supervision by constructing an objective function that aligns the hash code with the label matrix. The formula is as follows:

[0146] stCC T =I c×c B∈{-1,1};

[0147] Where B represents the binary hash code, B = [b1; b2; ...; b N ]∈{-1,1}N×k N represents the number of training data samples; k represents the length of the hash code; L represents the discrete label matrix, L∈{0,1} N×c ;c represents the total number of categories covered by all data;I c×c It is the identity matrix; the transformation matrix. Subject to CC T =I c×c The constraints ensure the orthogonality of matrix C;

[0148] S302. By introducing label information through the label matrix L, the embedded label information effectively maintains the similarity between and within modalities, achieving better information retrieval performance. Aligning the hash code with the label matrix can further enhance the preservation of semantic similarity. The aligned semantic information serves as guidance during training, helping to enhance the overall performance of multimodal hashing.

[0149] S303. Constructing a Tag Enhancement Strategy: Defining the Enhanced Tag Matrix This enhances label relevance; where L is a pre-labeled discrete multi-label matrix, L∈{0,1}. N×c L is associated with one or more labels, providing additional semantic information for multimodal data; in the label matrix L, element l ij The matrix M indicates whether the i-th sample belongs to the j-th class (1 for belonging, 0 for not belonging); it is an indicator matrix that records the relationship between each sample and each class, determining whether a sample belongs to a certain class; if a sample belongs to any class, the corresponding element is 1, otherwise it is 0; the matrix E is an adjustment matrix used to adjust the mapping process from discrete labels to continuous labels. The goal of matrix E is to measure the degree of adjustment required for each element in the original binary label matrix to align with the continuous label representation. This adjustment mapping aligns the discrete labels to the continuous label representation, specifically as follows: the slack variable ε is dynamically adjusted according to the class membership of each class; and by converting the original discrete binary labels into continuous and adjustable labels, the similarity between different classes can be captured more accurately, thus better capturing the intrinsic differences between classes;

[0150] S304, Enhanced Label Hash Learning: By capturing subtle differences between different categories, an enhanced label matrix is ​​used. The original label matrix L is replaced as input to better distinguish different categories and perform more accurate retrieval. The objective function for enhancing the label hash learning process is as follows:

[0151]

[0152] CC T =I c×c,B{-1,1};

[0153] Where α and γ are non-negative balancing parameters; c represents the total number of categories covered by all data. Enhanced tag matrix Regularization learning is used to prevent overfitting and improve the model's generalization ability.

[0154] S305. Fusion of Heterogeneous Multimodal Features: To effectively address the heterogeneity gap problem, heterogeneous multimodal features are encoded into a shared space, allowing direct comparison of data from different modalities. Specifically, dynamic modality weighting parameters fully consider the complementarity of multimodal features to improve the expressive and discriminative power of the learned hash codes. The objective function of the heterogeneous multimodal feature fusion process is as follows:

[0155]

[0156] in, It is a nonlinear embedding representation of the v-th modality, which captures the features of each modality by utilizing the relationship between randomly selected m anchor points and data samples. Matrix U v It is the orthogonal rotation matrix of the v-th mode. By mapping the nonlinear embedding of each mode to a common hash code space, the hash code of the specific features of each mode is effectively learned and preserved. To effectively capture the unique characteristics and advantages of each mode while taking into account the complementarity between modes, dynamic modal weighting parameters are introduced. This allows for flexible adjustment of the importance and weight contribution of each modality;

[0157] S306. Construction of the final objective function: The final objective function is obtained by integrating the objective function of the enhanced label hashing learning process and the objective function of the heterogeneous multimodal feature fusion process, as follows:

[0158]

[0159] stU v (U v ) T =I m×m CC T =I c×c B∈{-1,1};

[0160] in, Hash codes are learned by minimizing the reconstruction error between the nonlinear embedding and the learned hash, focusing on the nonlinear representation capability of the learned hash codes for the original data; The discrete labels are converted into continuous and adjustable labels. The purpose of this is to effectively distinguish the differences between various categories and provide more detailed semantic relationship information. By adjusting the elements in the label matrix, each sample can be more finely assigned to the corresponding category. Used to prevent overfitting and improve the system's generalization ability.

[0161] As attached Figure 5 As shown, the variable update optimization in step S4 of this embodiment obtains the enhanced label matrix for supervised hash code learning through an alternating optimization method. The orthogonal rotation matrix U used to map the nonlinear embedding of the v-th mode to a common hash code space v The binary hash code B is used for efficient retrieval, and the transformation matrix C is used to enhance semantic similarity preservation and fuse complementary features between different modalities; specifically as follows:

[0162] S401, Optimization and Update Specifically as follows:

[0163] S40101, in removing and After removing irrelevant terms, the objective function simplifies to

[0164] S40102, will Transforming the matrix into trace operation form helps to handle matrix operations more efficiently, reduce computational complexity, and improve computational efficiency; further... Find the partial derivative and then... Setting it to zero, we obtain the update function for the discrete label matrix.

[0165] S40103, By expanding the discrete label matrix The scope is expanded to further enhance the effectiveness of similarity supervision; and the discrete label matrix is ​​extended to... Specifically, the update result for the error category is restricted to be less than 0, while the update result for the true category should be greater than 1; thus adjusting... The specific optimization and update formula is as follows:

[0166]

[0167] S402, Optimized and updated U v The details are as follows:

[0168] Treating other variables as constants, the objective function is transformed into...

[0169] Among them, in U v (U v ) T =Im×m Under constraints, U v The optimal solution is transformed into solving The problem of maximizing the trace; the goal is to maximize Equivalent to maximizing Tr((U) v ) T G); U is derived using the Singular Value Decomposition (SVD) theorem. v The optimal solution is Among them, G l and G r These are the left and right singular vectors of G, respectively;

[0170] S403, Optimize and update C, as follows:

[0171] With other parameters fixed, the objective function for updating the transformation matrix C is expressed as:

[0172] Among them, in CC T =I c×c Under constraints, the objective function is transformed using the singular value decomposition principle. Transform it into a maximization problem, and use the singular value decomposition theorem to derive the optimal solution for C. Solve for the left and right singular vectors;

[0173] S404, Optimization and Update B, details are as follows:

[0174] Keeping all other variables constant, the objective function associated with variable B is:

[0175] Taking the partial derivative with respect to B and setting B to zero, we derive a closed-form solution for B as follows:

[0176] As attached Figure 6 As shown, in step S5 of this embodiment, the online query uses the hash code generated during the offline training phase to construct a hash function, which is used to generate hash codes for new query samples and obtain the linear projection matrix W for multimodal feature learning. Then, the symbolic function sgn() is used to obtain the binary hash code representations of the test dataset and the retrieval dataset samples respectively. The Hamming distance between the hash codes of the test dataset and the retrieval dataset samples is calculated. The retrieval dataset samples are sorted in ascending order according to the corresponding Hamming distance to obtain the final retrieval results. Finally, the mAP value is used to evaluate the retrieval performance. The details are as follows:

[0177] S501. Learning the hash code of out-of-sample query data: The hash function learning process is transformed into a classification problem. By concatenating the nonlinear embeddings of multimodal features, the projection matrix W is learned to represent the multimodal features. The specific objective function is as follows:

[0178]

[0179] in, Nonlinear embeddings representing multimodal features W is the projection matrix for multimodal feature learning. The objective function simplifies the training process and improves system efficiency and scalability; the formula for solving W is:

[0180]

[0181] S502. Generate hash codes for query samples: Use hash codes generated by hash functions for multimodal retrieval. The specific formula for predicting hash codes for the sample to be queried is as follows:

[0182]

[0183] Among them, X q This is a sample of query data. B q It is X q Hash code generated by a hash function; This represents the non-linear embedding of the query sample. n q This indicates the number of query samples; the linear projection matrix W is obtained during the training phase.

[0184] S503. Calculate Hamming distance: Select any test sample in the test dataset and obtain the Hamming distance between the hash code of the test sample and the hash codes of all samples in the retrieval dataset; sort the samples in the retrieval dataset according to their Hamming distances and store the relationship between the test sample distances and a set threshold.

[0185] If the distance between the test samples is less than the set threshold, they will be ranked higher.

[0186] If the distance to the test sample is greater than the set threshold, it will be ranked lower, thus verifying the accuracy of the retrieval.

[0187] For each modality, the average precision (AP) at different recall rates is calculated, and the average precision of each modality is averaged to obtain the mAP value of each modality.

[0188] Example 2:

[0189] As attached Figure 7 As shown, this embodiment provides a label-enhanced supervised multimodal hash retrieval system, which includes:

[0190] The data preprocessing unit is used to acquire and organize public datasets of image and text modalities, and divide each public dataset into training dataset, test dataset and retrieval dataset for subsequent use;

[0191] The feature extraction unit is used to extract features from the raw data of the public datasets of image modality and text modality respectively through a pre-trained network model, and obtain the deep features of image modality and text modality, which can more accurately reflect the underlying semantics of the data;

[0192] The offline training unit is used to train the objective function using image modal depth features and text modal depth features and their corresponding labels, and comprehensively considers the continuous and dynamic adjustment of discrete labels and the representation learning of optimized hash codes, thereby learning a reliable objective function.

[0193] The variable update and optimization unit is used to train and optimize each variable in the objective function using an alternating optimization method, thereby achieving the goal of comprehensively updating and optimizing the objective function;

[0194] The online query unit constructs a hash function using hash codes generated offline during training. This hash function generates hash codes for new query samples, and similarity retrieval is performed by calculating Hamming distance. The average retrieval performance is calculated through multiple iterations, and the mAP value is obtained to evaluate the retrieval performance. The mAP value is an average accuracy metric that measures the performance of tasks such as information retrieval or object detection.

[0195] The feature extraction unit in this embodiment includes:

[0196] The image modality feature extraction module is used to extract features from images using a ViT model pre-trained on the ImageNet dataset as an image encoder;

[0197] The text modality feature extraction module is used to extract features from images using the BERT model as a text encoder.

[0198] The offline training unit in this embodiment includes:

[0199] The similarity supervision optimization module is used to optimize the effectiveness of similarity supervision through the objective function. stCC T =I c×c Align the hash code with the label matrix if B∈{-1,1};

[0200] The tag enhancement strategy building module is used to define the enhanced tag matrix. To enhance the relevance of tags;

[0201] An enhanced label hashing learning module is used to enhance the label matrix by capturing small differences between different categories. Instead of the original label matrix L, it is used as input to better distinguish different categories and perform more accurate retrieval;

[0202] The heterogeneous multimodal feature fusion module is used to encode heterogeneous multimodal features into a shared space, allowing direct comparison of data from different modalities within the shared space;

[0203] The final objective function building module is used to obtain the final objective function by integrating the objective function of the enhanced label hash learning process and the objective function of the fusion heterogeneous multimodal feature process.

[0204] The variable update optimization unit in this embodiment includes:

[0205] Optimize and update module one, used to remove and After removing irrelevant terms, simplify the objective function, and then transform the simplified objective function into trace form before performing further operations. Find the partial derivative and then... Setting it to zero yields the optimized update function for the discrete label matrix.

[0206] The second optimization module is used to transform the objective function and quantitatively optimize and update U using singular value decomposition, while treating other variables as constants. v ;

[0207] The third optimization module is used to transform the objective function into a maximization problem when other variables are considered constant, and to quantitatively optimize and update C using singular value decomposition.

[0208] The fourth optimization and update module is used to obtain the closed-form solution of B by taking the partial derivative of the objective function associated with variable B with respect to B, and setting B to zero, while treating other variables as constants, thereby completing the optimization and update of B.

[0209] As attached Figure 8As shown, the system's workflow mainly consists of five parts: data preprocessing, deep feature representation, offline training, variable update and optimization, and online querying. Data preprocessing involves downloading public datasets containing both image and text modalities, further dividing each dataset into training, testing, and retrieval datasets as the raw data input to the model. Deep feature extraction is responsible for extracting features from the raw data received by the model, using pre-trained ViT and BERT models to extract image and text features respectively. These extracted features serve as data for the offline training phase. Offline training involves training the model using a label enhancement strategy. In this phase, hyperparameters, the number of anchor points, and the number of bits in the binary hash code are initialized. Next, a label enhancement strategy is constructed to train the model, transforming discrete labels into continuous and adjustable labels to better distinguish different categories and provide more detailed semantic relationship information. Subsequently, a subset of training samples is randomly selected as anchor points to construct a similarity representation matrix for the image and text modalities based on the extracted deep feature data. With the enhanced label information, the similarity representation matrix helps the model more accurately capture the semantic differences between different categories. Finally, an orthogonal rotation matrix U is used... v The features of each modality are mapped to a common hash code space. By adjusting the weight parameters of each modality, the importance and contribution of each modality are flexibly adjusted, achieving feature fusion between the two modalities. This process fully utilizes the complementarity of multimodal data, effectively preserving the specific features of each modality, and thus learning a high-quality hash code. The variable update optimization task involves training and optimizing each variable in the objective function using an alternating optimization method to comprehensively optimize the overall objective function. In each iteration, by keeping other variables fixed, each variable is optimized one by one, which simplifies the objective function optimization process and improves the computational efficiency of the model. In the online query phase, the optimal binary hash code obtained in the offline training phase guides the construction of a target multimodal hash function to generate a hash code of a specified number of bits for out-of-sample query data. Subsequently, similarity search is performed by calculating Hamming distance. The average retrieval performance is calculated through multiple iterations of the above steps, and the mAP value is obtained to evaluate the retrieval performance.

[0210] Example 3:

[0211] This embodiment also provides an electronic device, including: a memory and a processor;

[0212] The memory stores the instructions executed by the computer.

[0213] The processor executes computer execution instructions stored in the memory, causing the processor to perform the label-enhanced supervised multimodal hash retrieval method according to any embodiment of the present invention.

[0214] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0215] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0216] Example 4:

[0217] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the label-enhanced supervised multimodal hash retrieval method according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0218] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0219] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0220] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0221] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A supervised multimodal hash retrieval method based on label enhancement, characterized in that, The method is as follows: Data preprocessing: Obtain and organize public datasets of image and text modalities, and divide each public dataset into training dataset, test dataset, and retrieval dataset; Extracting deep features from data: Using a pre-trained network model, features are extracted from the raw data of the public datasets for both image and text modalities to obtain deep features for both image and text modalities. Offline training: The objective function is trained using image modality depth features and text modality depth features and their corresponding labels. The continuous and dynamic adjustment of discrete labels and the representation learning of optimized hash codes are also considered to learn the objective function. Variable update optimization: The alternating optimization method is used to train and optimize each variable in the objective function, thereby achieving the goal of comprehensively updating and optimizing the objective function; Online query: A hash function is constructed using hash codes generated offline. Hash codes are generated for new query samples using the hash function. Similarity retrieval is then performed by calculating Hamming distance. The average retrieval performance is calculated through multiple iterations, and the mAP value is obtained to evaluate the retrieval performance. The offline training process is detailed below: To optimize the effectiveness of similarity supervision, an objective function is constructed that aligns the hash code with the label matrix, as shown in the formula: ； in, Represents a binary hash code, ; Indicates the number of training data samples; Indicates the length of the hash code; Represents a discrete label matrix. ; This indicates the total number of categories covered by all data; It is the identity matrix; the transformation matrix. Received Constraints to ensure the matrix Orthogonality; Constructing a tag enhancement strategy: By defining the enhanced tag matrix To enhance tag relevance; among which, It is a pre-labeled discrete multi-label matrix. , Associating with one or more labels provides additional semantic information for multimodal data; in the label matrix In, elements ij Indicates the first Does the _ sample belong to the _ ... Class; Matrix It is an indicator matrix used to record the relationship between each sample and each class, determining whether a sample belongs to a certain class; if a sample belongs to any class, the corresponding element is 1, otherwise it is 0; the matrix It is an adjustment matrix used to adjust the mapping process from discrete labels to continuous labels. The goal is to measure the degree of adjustment required to align each element in the original binary label matrix with the continuous label representation. This degree of adjustment maps discrete labels to continuous label representations, specifically by dynamically adjusting slack variables based on the class membership of each category. By converting the original discrete binary labels into continuous and adjustable labels, it captures the similarity between different categories, thereby better capturing the intrinsic differences between categories; Augmented label hashing learning: By capturing the differences between different categories, an augmented label matrix is ​​used. Replace the original label matrix The objective function for the augmented label hash learning process, as input, is as follows: ； ； in, and It is a non-negative equilibrium parameter; This indicates the total number of categories covered by all data; Enhanced tag matrix Regularization learning is used to prevent overfitting and improve the model's generalization ability. Fusion of heterogeneous multimodal features: Heterogeneous multimodal features are encoded into a shared space, allowing direct comparison of data from different modalities. Dynamic modality weighting parameters are used to fully consider the complementarity of multimodal features, thereby improving the expressive and discriminative power of the learned hash codes. The objective function of the heterogeneous multimodal feature fusion process is as follows: ； in, It is the first The nonlinear embedding representation of each modality is achieved by utilizing randomly selected... The relationship between anchor points and data samples captures the features of each modality. ;matrix It is the first The orthogonal rotation matrix of each mode, by mapping the nonlinear embedding of each mode to a common hash code space, effectively learns and preserves the hash code of each mode's specific features. Dynamic modal weighting parameters This allows for flexible adjustment of the importance and weight contribution of each modality; Construction of the final objective function: The final objective function is obtained by integrating the objective function of the enhanced label hashing learning process and the objective function of the heterogeneous multimodal feature fusion process, as follows: ； ； in, Hash codes are learned by minimizing the reconstruction error between the nonlinear embedding and the learned hash, focusing on the nonlinear representation capability of the learned hash codes for the original data; This transforms discrete labels into continuous and adjustable labels, while providing more detailed semantic relationship information. By adjusting the elements in the label matrix, each sample can be more precisely assigned to the corresponding category. Used to prevent overfitting and improve the system's generalization ability.

2. The supervised multimodal hash retrieval method based on tag enhancement according to claim 1, characterized in that, The data preprocessing is as follows: We obtained and downloaded public datasets containing image and text modalities from public network resources as the original datasets; these public datasets included the MIR Flickr dataset, the NUS-WIDE dataset, and the MS COCO dataset. The downloaded raw dataset is partitioned as follows: Randomly select a subset of image-text pairs from the MIR Flickr dataset, NUS-WIDE dataset, and MS COCO dataset as the test set to evaluate test performance; Excluding samples from the test set, the remaining image-text pairs in the MIR Flickr dataset, NUS-WIDE dataset, and MS COCO dataset are used as the retrieval dataset for retrieval operations; In the retrieval dataset, any portion of image-text pairs is randomly selected as the training set for training the objective function.

3. The label-enhanced supervised multimodal hash retrieval method according to claim 1 or 2, characterized in that, The specific steps for extracting deep features from the data are as follows: Image modal depth feature extraction: A ViT model pre-trained on the ImageNet dataset is used as the image encoder to extract features from the image; specifically as follows: The input image will be uniformly adjusted to Pixel size, obtain the reshaped image; Convert the reshaped image into a flattened two-dimensional planar image block; Introducing position embedding to capture spatial relationships between image patches; Feature extraction is performed by stacking multiple Transformer layers. Each Transformer layer includes a self-attention mechanism and a fully connected feedforward network. Global average pooling is performed on the output of all positions of the last Transformer layer to obtain the global average pooling representation of the entire image. Linear projection is performed on the global average pooling representation of the entire image to obtain the image's embedding feature representation, which is a vector containing 4096-dimensional features. Extracting deep features from text modalities: Using the BERT model as a text encoder, features are extracted from the text modalities; specifically as follows: The input text is divided into tokens, and each token is transformed into a vector representation through an embedding layer; Introducing positional encoding, which considers the positional information of words in a sentence; By using a multi-head self-attention mechanism network, different parts are focused on at different positions in the input sequence to capture contextual relationships, and a feedforward neural network is applied after each attention layer to perform nonlinear transformation on the features; After multi-head self-attention and feedforward neural network operations, layer normalization is applied to obtain the deep feature representation of the text. Among them, the feature extraction dimension values ​​of the text data are different on three different datasets: MIR Flickr dataset, NUS-WIDE dataset and MS COCO dataset.

4. The supervised multimodal hash retrieval method based on tag enhancement according to claim 1, characterized in that, Variable update optimization obtains the enhanced label matrix for supervised hash code learning through an alternating optimization method. Used to transfer the first The nonlinear embeddings of each modality are mapped to an orthogonal rotation matrix in a common hash code space. Binary hash codes for efficient retrieval And a transformation matrix for enhancing semantic similarity preservation and fusing complementary features between different modalities. The details are as follows: Optimization and Update The details are as follows: In removal and After removing irrelevant terms, the objective function simplifies to ; Will Convert to trace operation form; then... Find the partial derivative and then... Setting it to zero, we obtain the update function for the discrete label matrix. ; By expanding the discrete label matrix The scope is expanded to further enhance the effectiveness of similarity supervision; and the discrete label matrix is ​​extended to... In this context, the update result for the error category is restricted to be less than 0, while the update result for the true category should be greater than 1; thus adjusting... The specific optimization and update formula is as follows: ； Optimization and Update The details are as follows: Treating other variables as constants, the objective function is transformed into... ; Among them, Under constraints, The optimal solution is transformed into solving The problem of maximizing the trace; the goal is to maximize This is equivalent to maximizing ; derived using the singular value decomposition theorem The optimal solution is ;in, and They are The left singular vector and the right singular vector; Optimization and Update The details are as follows: With other parameters fixed, the transformation matrix The update objective function is expressed as ; Among them, Under constraints, the objective function is transformed using the singular value decomposition principle. Transform it into a maximization problem, and derive it using the singular value decomposition theorem. The optimal solution is obtained through Solve for the left and right singular vectors; Optimization and Update The details are as follows: Keeping all other variables constant, and the variable The associated objective function is ; right Find the partial derivative and then... Set the value to zero, and derive... The closed solution is .

5. The supervised multimodal hash retrieval method based on tag enhancement according to claim 1, characterized in that, Online queries utilize hash codes generated during the offline training phase to construct a hash function, which is used to generate hash codes for new query samples and obtain a linear projection matrix for multimodal feature learning. Then, the symbolic function sgn() is used to obtain the binary hash code representations of the test dataset and the retrieval dataset samples respectively. The Hamming distance between the hash codes of the test dataset and the retrieval dataset samples is calculated. The retrieval dataset samples are sorted in ascending order according to the corresponding Hamming distance to obtain the final retrieval results. Finally, the mAP value is used to evaluate the retrieval performance; the details are as follows: Learning the hash code of out-of-sample query data: The hash function learning process is transformed into a classification problem. By concatenating the nonlinear embeddings of multimodal features, the projection matrix is ​​learned to represent the hash function. To perform multimodal feature learning; the specific objective function is as follows: ； in, Nonlinear embeddings representing multimodal features ; It is the projection matrix for multimodal feature learning. ; The solution formula is: ； Generate hash codes for query samples: Multimodal retrieval is performed using hash codes generated by a hash function. The specific formula for predicting the hash code for the query sample is as follows: ； in, This is a sample of query data. ; yes Hash code generated by a hash function; This represents the non-linear embedding of the query sample. ; Represents the number of query samples; linear projection matrix Acquired during the training phase; Calculate Hamming distance: Select any test sample in the test dataset and obtain the Hamming distance between the hash code of the test sample and the hash codes of all samples in the retrieval dataset; sort the samples in the retrieval dataset according to their Hamming distances and store the relationship between the test sample distances and a set threshold. If the distance between the test samples is less than the set threshold, they will be ranked higher. If the distance to the test sample is greater than the set threshold, it will be ranked lower, thus verifying the accuracy of the retrieval. For each modality, calculate the average precision under different recall rates, and average the average precision of each modality to obtain the mAP value of each modality.

6. A supervised multimodal hash retrieval system based on label enhancement, characterized in that, The system includes: The data preprocessing unit is used to acquire and organize public datasets of image and text modalities, and divide each public dataset into training dataset, test dataset and retrieval dataset; The feature extraction unit is used to extract features from the raw data of the public datasets of image modality and text modality respectively through a pre-trained network model, and obtain the image modality depth features and the text modality depth features. The offline training unit is used to train the objective function using image modal depth features and text modal depth features and their corresponding labels, and comprehensively considers the continuous and dynamic adjustment of discrete labels and the representation learning of optimized hash codes, thereby learning the objective function. The variable update and optimization unit is used to train and optimize each variable in the objective function using an alternating optimization method, thereby achieving the goal of comprehensively updating and optimizing the objective function; The online query unit is used to construct a hash function using hash codes generated offline, generate hash codes for new query samples using the hash function, perform similarity retrieval by calculating Hamming distance, calculate the average retrieval performance through multiple iterations, and obtain the mAP value to evaluate the retrieval performance. The offline training unit works as follows: To optimize the effectiveness of similarity supervision, an objective function is constructed that aligns the hash code with the label matrix, as shown in the formula: ； in, Represents a binary hash code, ; Indicates the number of training data samples; Indicates the length of the hash code; Represents a discrete label matrix. ; This indicates the total number of categories covered by all data; It is the identity matrix; the transformation matrix. Received Constraints to ensure the matrix Orthogonality; Constructing a tag enhancement strategy: By defining the enhanced tag matrix To enhance tag relevance; among which, It is a pre-labeled discrete multi-label matrix. , Associating with one or more labels provides additional semantic information for multimodal data; in the label matrix In, elements ij Indicates the first Does the _ sample belong to the _ ... Class; Matrix It is an indicator matrix used to record the relationship between each sample and each class, determining whether a sample belongs to a certain class; if a sample belongs to any class, the corresponding element is 1, otherwise it is 0; the matrix It is an adjustment matrix used to adjust the mapping process from discrete labels to continuous labels. The goal is to measure the degree of adjustment required to align each element in the original binary label matrix with the continuous label representation. This degree of adjustment maps discrete labels to continuous label representations, specifically by dynamically adjusting slack variables based on the class membership of each category. By converting the original discrete binary labels into continuous and adjustable labels, it captures the similarity between different categories, thereby better capturing the intrinsic differences between categories; Augmented label hashing learning: By capturing the differences between different categories, an augmented label matrix is ​​used. Replace the original label matrix The objective function for the augmented label hash learning process, as input, is as follows: ； ； in, and It is a non-negative equilibrium parameter; This indicates the total number of categories covered by all data; Enhanced tag matrix Regularization learning is used to prevent overfitting and improve the model's generalization ability. Fusion of heterogeneous multimodal features: Heterogeneous multimodal features are encoded into a shared space, allowing direct comparison of data from different modalities. Dynamic modality weighting parameters are used to fully consider the complementarity of multimodal features, thereby improving the expressive and discriminative power of the learned hash codes. The objective function of the heterogeneous multimodal feature fusion process is as follows: ； in, It is the first The nonlinear embedding representation of each modality is achieved by utilizing randomly selected... The relationship between anchor points and data samples captures the features of each modality. ;matrix It is the first The orthogonal rotation matrix of each mode, by mapping the nonlinear embedding of each mode to a common hash code space, effectively learns and preserves the hash code of each mode's specific features. Dynamic modal weighting parameters This allows for flexible adjustment of the importance and weight contribution of each modality; Construction of the final objective function: The final objective function is obtained by integrating the objective function of the enhanced label hashing learning process and the objective function of the heterogeneous multimodal feature fusion process, as follows: ； ； in, Hash codes are learned by minimizing the reconstruction error between the nonlinear embedding and the learned hash, focusing on the nonlinear representation capability of the learned hash codes for the original data; This transforms discrete labels into continuous and adjustable labels, while providing more detailed semantic relationship information. By adjusting the elements in the label matrix, each sample can be more precisely assigned to the corresponding category. Used to prevent overfitting and improve the system's generalization ability.

7. The label-enhanced supervised multimodal hash retrieval system according to claim 6, characterized in that, The feature extraction unit includes: The image modality feature extraction module is used to extract features from images using a ViT model pre-trained on the ImageNet dataset as an image encoder; The text modality feature extraction module is used to extract features from images using the BERT model as a text encoder; The offline training unit includes: The similarity supervision optimization module is used to optimize the effectiveness of similarity supervision through the objective function. Align the hash codes with the label matrix; The tag enhancement strategy building module is used to define the enhanced tag matrix. To enhance the relevance of tags; An enhanced label hashing learning module is used to enhance the label matrix by capturing the gaps between different categories. Replace the original label matrix This serves as input, allowing for better differentiation of different categories and more accurate retrieval. The heterogeneous multimodal feature fusion module is used to encode heterogeneous multimodal features into a shared space, allowing direct comparison of data from different modalities within the shared space; The final objective function construction module is used to obtain the final objective function by integrating the objective function of the enhanced label hashing learning process and the objective function of the fusion heterogeneous multimodal feature process; The variable update optimization unit includes: Optimize and update module one, used to remove and After removing irrelevant terms, simplify the objective function, and then transform the simplified objective function into trace form before performing further operations. Find the partial derivative and then... Setting it to zero yields the optimized update function for the discrete label matrix. ; The second optimization and update module is used to transform the objective function and quantitatively optimize and update it using singular value decomposition, while treating other variables as constants. ; The third optimization module transforms the objective function into a maximization problem when other variables are considered constant, and uses singular value decomposition to quantitatively optimize and update the function. ; Optimize and update module four, used to utilize variables when other variables are considered constants. Related objective function pairs Find the partial derivative and then... Set to zero, get The closed solution is obtained, thus completing the process. Optimization and updates.

8. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the label-enhanced supervised multimodal hash retrieval method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the label-enhanced supervised multimodal hash retrieval method as described in any one of claims 1 to 5.

Images