Noise tag multi-mode hash retrieval method, system and equipment based on semantic alignment enhancement and medium

Abstract

The invention discloses a noise tag multi-modal Hash retrieval method, system and device based on semantic alignment enhancement and a medium, belongs to the field of artificial intelligence and the technical field of multimedia information retrieval, and aims to solve the technical problem of how to avoid retrieval performance reduction caused by noise tags in the multi-modal Hash retrieval process. According to the technical scheme, the method comprises the following steps of: constructing a social media multi-modal data set; constructing a noise label multi-modal Hash retrieval model based on semantic alignment enhancement; constructing the noise label multi-modal Hash retrieval model based on semantic alignment enhancement based on a neural network and a deep learning technology, specifically, multi-modal fusion; semantic alignment enhancement of noise perception is carried out; performing hash retrieval query; and training the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and multi-modal hash retrieval technology, in particular to a multi-modal hash retrieval method, system, device and medium based on semantic alignment enhancement of noise labels. BACKGROUND

[0002] With the explosive growth of multi-source heterogeneous data such as images, texts, audios, etc., multi-modal information retrieval has become a key technology to support intelligent search, content recommendation and cross-modal understanding. Hash retrieval maps high-dimensional multi-modal features into compact binary hash codes, greatly reducing storage overhead and accelerating similarity calculation while striving to maintain cross-modal semantic consistency, and is widely used in large-scale multi-modal retrieval systems.

[0003] However, most existing multi-modal hash methods rely heavily on accurate labeled supervision information to learn hash functions with strong semantic discriminability. In actual application scenarios, due to high artificial labeling cost or inherent noise in automatic labeling, training data often contains a large number of incorrect or unreliable labels, i.e. noise labels. Such noise can seriously mislead the learning process of hash codes, resulting in semantic misalignment of generated hash representations and significant decline in retrieval performance. To alleviate the impact of noisy labels, some methods attempt to introduce label correction or loss modification mechanisms, but face two major challenges in the multi-modal scenario: first, the feature distributions of different modalities differ greatly, and if not effectively aligned, the negative impact of noisy labels will be further amplified; second, existing methods lack explicit modeling of semantic structure, making it difficult to maintain intra-class compactness and inter-class separability under noise interference, resulting in insufficient discriminability of hash codes.

[0004] Although recent studies have explored noise-robust hash learning, they are mostly limited to single-modal scenarios or only handle multi-modal data through simple joint training, failing to fully exploit multi-modal complementarity to enhance tolerance to noisy labels. More importantly, existing methods lack a unified framework that can dynamically identify reliable samples, adaptively optimize semantic supervision signals, and explicitly align class semantic structures in the hash space, making it difficult to achieve efficient and accurate multi-modal retrieval in a strong noise environment.

[0005] Therefore, how to avoid the decline in retrieval performance caused by noisy labels in the process of multi-modal hash retrieval, ensure the acquisition of noise-robust, semantically discriminative hash codes, and improve retrieval efficiency is a technical problem to be solved. SUMMARY

[0006] The technical task of the present application is to provide a noise label multi-modal hash retrieval method, system, device and medium based on semantic alignment enhancement, to solve the problem of how to avoid the decline of retrieval performance caused by noise labels in the process of multi-modal hash retrieval, ensure the acquisition of noise-robust and semantically discriminative hash codes, and improve the retrieval efficiency.

[0007] The technical task of the present application is achieved in the following way: a noise label multi-modal hash retrieval method based on semantic alignment enhancement, which is as follows: Constructing a social media multi-modal dataset: obtaining and downloading a public dataset containing an image modal dataset and a text modal dataset from public network resources as an original dataset, standardizing the image modal dataset and the text modal dataset respectively, and extracting deep image features and deep text features from the standardized image modal dataset and the standardized text modal dataset respectively through a BLIP-3 pre-training model, constructing a plurality of samples according to the deep image features, the deep text features and the original noise labels carried by the original dataset, the sample form being (deep image features, deep text features, original noise labels), and constructing a final dataset containing a training set, a test set and a retrieval set according to the original noise label categories and the number of samples; Constructing a noise label multi-modal hash retrieval model based on semantic alignment enhancement: constructing a noise label multi-modal hash retrieval model based on semantic alignment enhancement based on neural network and deep learning technology, specifically: multi-modal fusion: aligning the deep image features and the deep text features, concatenating and fusing the deep image features and the deep text features after feature alignment to generate a feature fusion matrix, and obtaining hash codes through a hash mapping network; noise-aware semantic alignment enhancement: combining the feature fusion matrix with the original noise labels, dynamically identifying reliable samples through an adaptive label optimization process, generating clean soft pseudo labels, and then constructing class-level semantic prototypes based on the clean soft pseudo labels, and aligning and optimizing the hash codes of each sample and the class-level semantic prototypes corresponding to the hash codes of each sample in the hash space to obtain noise-robust hash codes; hash retrieval query: performing similarity retrieval on the noise-robust hash codes through Hamming distance, multiple iterations and average precision calculation, and the average precision is used to evaluate the retrieval performance; Training the model: training the noise label multi-modal hash retrieval model based on semantic alignment enhancement through the final dataset to obtain the trained noise label multi-modal hash retrieval model based on semantic alignment enhancement.

[0008] As a preferred embodiment, the deep image features are obtained as follows: Preprocessing image modal data: the images in the image modal dataset are uniformly cropped and reshaped to HxH pixel size, and the cropped and reshaped images are normalized to make the pixel value range of the cropped and reshaped images between [0, 1], eliminating brightness and contrast differences; Extracting deep image features: deep image features are extracted through the BLIP-3 pre-training model to capture subtle changes in the image, and then deep image features are obtained for subsequent analysis; The deep text feature is obtained as follows: Preprocessing text modal data: text preprocessing operations such as text cleaning, text segmentation, and text vectorization are performed on the text in the text modal data, and the text information is mapped to a text vector; Extracting deep text features: deep text features are extracted through the BLIP-3 pre-training model to capture detailed text descriptions, ensuring that the captured detailed text descriptions reflect the complexity of the text content, and the captured detailed text descriptions are standardized to obtain deep text features for model learning and analysis.

[0009] More preferably, the multi-modal fusion is as follows: Feature alignment: aligned deep image features and aligned deep text features are obtained through a feature aligner composed of multiple linear layers and activation functions, as follows: ; ; wherein, and represent deep image features and deep text features, respectively; and represent image feature aligner and text feature aligner, respectively; and represent training parameters; and represent aligned deep image features and aligned deep text features, respectively; Feature concatenation fusion: the aligned deep image features and the aligned deep text features are subjected to a feature concatenation fusion function to generate a feature fusion matrix, as follows: ; wherein, represents the feature concatenation fusion function; represents the feature fusion matrix; and represent aligned deep image features and aligned deep text features, respectively; represents the concatenation operation; Hash mapping: The feature fusion matrix is ​​processed by a hash mapping function to generate a relaxed hash code, and then processed... The function generates hash codes using the following formula: ; ; in, Represents a hash mapping function; Indicates the first The relaxed hash code of each sample; Indicates the first The hash code of each sample; Indicates the first Feature fusion matrix of each sample; This represents the training parameters.

[0010] More specifically, the noise-aware semantic alignment enhancement is as follows: Adaptive label optimization: The contrastive loss for each sample is calculated by combining the relaxed hash code and the original noisy label, as shown in the following formula: ; ; ; in, Indicates the first Contrast loss for each sample; Indicates the first The semantic center of a class; Indicates sample Category semantic confidence; Represents the contrast mask matrix; Indicates the original noise label; Generate adaptive clean soft pseudo-labels: Fit the contrastive loss of the samples to a bimodal Gaussian mixture model, calculate the posterior probability of a sample belonging to a clean label, and generate clean soft pseudo-labels based on an adaptive threshold, as shown in the following formula: ; ; in, Indicates the first The posterior probability of a sample; Indicates normalized contrast loss ; Indicates a clean, soft-sell label; Indicates an indicator function; Represents the mathematical expectation; Indicates variance; Indicates the number of samples; Prototype-guided semantic alignment enhancement: utilize clean soft pseudo labels for classes Construct semantic prototypes and dynamically update them in training, as follows: ; ; ; wherein, represents the positive class indicated by the clean soft pseudo label; represents the semantic prototype; is the cosine distance; is the temperature coefficient; is the numerical stability term; is the number of samples of the clean soft pseudo label; represents the number of classes.

[0011] More preferably, the hash retrieval query is as follows: Calculate the average precision, as follows: ; wherein, represents the total number of samples in the retrieval set associated with the query sample in the test set; is the index upper bound of the sample; represents the precision of the first instances related, when the sample in the retrieval set is related to the query sample in the test set , otherwise ; Calculate the average precision mean, and average the average precision mean by calculating the average precision mean at different recall rates to obtain the MAP value, as follows: ; wherein, represents the number of test sets; represents the value of the th .

[0012] More preferably, the training model is as follows: Construct the loss function: the goal of the noise label multi-modal hash retrieval model based on semantic alignment enhancement is to utilize the feature fusion matrix and the clean soft pseudo label, and train an efficient multi-modal hash process through the relaxed hash code and the hash code, finally learn a semantic discriminative and noise-resistant multi-modal hash retrieval, utilize the feature fusion matrix , the clean soft pseudo label , the relaxed hash code and the hash code The noise label multi-modal hash retrieval model based on semantic alignment enhancement is trained by constructing a loss function; specifically as follows: The similarity preservation loss is constructed, and the formula is as follows: ; ; wherein, , denotes the relaxed hash code; denotes the clean soft pseudo label; denotes the cosine similarity; is the number of samples of the clean soft pseudo label; denotes the reconstruction similarity; The quantization loss is constructed, and the formula is as follows: ; wherein, denotes the relaxed hash code; denotes the hash code; Optimization model: using Adam algorithm as the optimization function of the noise label multi-modal hash retrieval model based on semantic alignment enhancement, the model parameters are iteratively updated by minimizing the constructed loss function; wherein, the learning rate parameter is set to 0.001, and other hyperparameters use the default value in PyTorch.

[0013] A noise label multi-modal hash retrieval system based on semantic alignment enhancement, which is used to realize the noise label multi-modal hash retrieval method based on semantic alignment enhancement as described above; the system comprises: A data set construction unit is configured to obtain and download a public data set containing an image modal data set and a text modal data set as an original data set from a public network resource, standardize the image modal data set and the text modal data set respectively, and extract deep image features and deep text features from the standardized image modal data set and the standardized text modal data set by using a BLIP-3 pre-training model, construct a plurality of samples according to the deep image features, the deep text features and the original noise label carried by the original data set, the sample form is (deep image features, deep text features, original noise label), and construct a final data set containing a training set, a test set and a retrieval set according to the original noise label category and the sample number; The model construction unit is configured to: perform feature alignment on the depth image features and the depth text features based on a neural network and a deep learning technology, generate a feature fusion matrix by splicing and fusing the depth image features and the depth text features after the feature alignment, and obtain a hash code through a hash mapping network; dynamically identify reliable samples through an adaptive label optimization process by combining the feature fusion matrix with original noise labels, generate clean soft pseudo labels, and then construct a class-level semantic prototype based on the clean soft pseudo labels, and perform alignment optimization on the hash code of each sample and the class-level semantic prototype corresponding to the hash code of each sample in a hash space to obtain a noise-resistant robust hash code; and perform similarity retrieval on the noise-resistant robust hash codes through a Hamming distance, and calculate average precision through multiple iterations, where the average precision is used to evaluate retrieval performance. The model training module is configured to train the noise label multi-modal hash retrieval model based on semantic alignment enhancement through the final data set to obtain the trained noise label multi-modal hash retrieval model based on semantic alignment enhancement.

[0014] Preferably, the model construction unit comprises: The feature alignment module is configured to obtain the aligned depth image features and the aligned depth text features through a feature aligner composed of multiple linear layers and activation functions, and the formula is as follows: ; ; wherein, and represent the depth image features and the depth text features, respectively; and represent the image feature aligner and the text feature aligner, respectively; and represent training parameters; and represent the aligned depth image features and the aligned depth text features, respectively; The feature splicing and fusion module is configured to generate a feature fusion matrix by performing a feature splicing and fusion function on the aligned depth image features and the aligned depth text features, and the formula is as follows: ; wherein, represents the feature splicing and fusion function; represents the feature fusion matrix; and represent the aligned depth image features and the aligned depth text features, respectively; represents a splicing operation; The hash mapping module is configured to generate a relaxed hash code by performing a hash mapping function on the feature fusion matrix, and then perform a hash code optimization process on the relaxed hash code to obtain the noise-resistant robust hash code. The function generates hash codes using the following formula: ; ; in, Represents a hash mapping function; Indicates the first The relaxed hash code of each sample; Indicates the first The hash code of each sample; Indicates the first Feature fusion matrix of each sample; Indicates training parameters; The adaptive label optimization module combines the relaxed hash code and the original noisy label to calculate the contrastive loss for each sample, as shown in the following formula: ; ; ; in, Indicates the first Contrast loss for each sample; Indicates the first The semantic center of a class; Indicates sample Category semantic confidence; Represents the contrast mask matrix; Indicates the original noise label; The clean soft pseudo-label generation module is used to fit the contrastive loss of the samples to a bimodal Gaussian mixture model, calculate the posterior probability of a sample belonging to a clean label, and generate clean soft pseudo-labels based on an adaptive threshold, as shown in the following formula: ; ; in, Indicates the first The posterior probability of a sample; Indicates normalized contrast loss ; Indicates a clean, soft-sell label; Indicates an indicator function; Represents the mathematical expectation; Indicates variance; Indicates the number of samples; A prototype-guided semantic alignment enhancement module for using clean soft pseudo-labels for categories. Construct a semantic prototype and update it dynamically during training, as shown in the following formula: ; ; ; in, A positive class indicating a clean soft pseudo-label; Represents the semantic prototype; Cosine distance; Temperature coefficient; It is a numerically stable term; The number of samples with clean soft fake labels; Indicates the number of categories; The Average Precision (MAP) calculation module is used to calculate the mean precision under different recall rates, and then average the mean precision to obtain the MAP value. The formula is as follows: ; ; in, This represents the total number of samples in the retrieval set that are associated with the query samples in the test set; This is the upper bound of the sample index; Indicates the relevant preceding The precision of the nth instance, when the retrieval set When a sample is relevant to a query sample in the test set ,otherwise ; Indicates the number of test sets; Indicates the first indivual value.

[0015] An electronic device includes: a memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the noise tag multimodal hash retrieval method based on semantic alignment enhancement as described above.

[0016] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the semantic alignment-enhanced noise tag multimodal hash retrieval method described above.

[0017] The semantic alignment-enhanced noise tag multimodal hash retrieval method, system, device, and medium of the present invention have the following advantages: (i) This invention introduces a noise-aware adaptive label optimization mechanism for the first time in multimodal hash retrieval, which can dynamically identify reliable samples from the original noisy labels and generate clean soft pseudo labels, effectively mitigating the negative impact of noisy labels on hash learning and significantly improving the robustness and generalization ability of the model in real noise scenarios. (ii) This invention uses BLIP-3 as a deep feature extraction encoder for modalities, which not only enriches the semantic information of multimodal data, but also more accurately captures the underlying semantic information of images and text, thereby enriching the feature semantic information of multimodal data and improving the accuracy and efficiency of retrieval. (III) This invention proposes a semantic alignment enhancement strategy guided by category prototypes, which explicitly models intra-class compactness and inter-class separability in the hash space. By bringing the sample hash code closer to its positive class prototype and pushing it further away from its negative class prototype, the semantic discrimination ability of the hash code is enhanced, thereby improving the accuracy of cross-modal retrieval. (iv) This invention reconstructs the semantic similarity between samples through soft pseudo-labels, effectively capturing the high-level semantic consistency of multimodal data, overcoming the problem of misjudgment of similarity caused by noisy labels in traditional methods, and enhancing the reliability of cross-modal semantic alignment; (v) The prototype alignment mechanism of the present invention has dynamic update capability. The category prototypes adapt and evolve with the training process, which can continuously reflect the current optimal semantic structure, avoid semantic bias caused by static prototypes, and further improve the stability and discriminability of hash representation. (vi) This invention does not rely on clean labels or additional manual annotation. It can learn high-quality hash functions using only noisy supervision information, which greatly reduces the dependence on the quality of labeled data. It is suitable for large-scale weak supervision or multi-source crawling multimodal data scenarios. (vii) The present invention designs an end-to-end joint optimization framework that organically integrates similarity preservation loss, prototype alignment loss and quantization loss, and collaboratively optimizes multimodal feature representation and hash code generation, thereby reducing quantization error while preserving semantic consistency and achieving efficient and high-precision hash retrieval. (viii) This invention has good scalability and versatility. Its noise perception and semantic alignment mechanism can be flexibly adapted to different hash network structures and multimodal data types, providing an efficient, robust and practical technical solution for large-scale, weakly supervised, cross-modal retrieval applications. (ix) In the process of multimodal hash retrieval, the present invention effectively ensures that the output hash code is noise-resistant, robust, and semantically discriminative. While ensuring retrieval efficiency, it significantly improves the generalization and retrieval performance of the model in real noisy label scenarios and avoids the problem of retrieval performance degradation caused by noisy labels. Attached Figure Description

[0018] The invention will be further described below with reference to the accompanying drawings.

[0019] Appendix Figure 1 The flowchart shows the multimodal hash retrieval method for noise labels based on semantic alignment enhancement. Appendix Figure 2 A flowchart for constructing a multimodal dataset; Appendix Figure 3 A flowchart for constructing a noisy label multimodal hash retrieval model based on semantic alignment enhancement; Appendix Figure 4 A flowchart illustrating the process of multimodal fusion; Appendix Figure 5 Flowchart for noise-aware semantic alignment enhancement; Appendix Figure 6 This is a flowchart of a hash retrieval query. Appendix Figure 7 This is a flowchart of the training model. Detailed Implementation

[0020] The semantic alignment-enhanced noisy tag multimodal hash retrieval method, system, device, and medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this disclosure. In the following detailed description, numerous specific details are set forth to provide a comprehensive understanding of the embodiments of this disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of this disclosure.

[0021] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse.

[0022] Example 1:

[0023] As attached Figure 1 As shown in the figure, this embodiment provides a noise tag multimodal hash retrieval method based on semantic alignment enhancement, which is as follows: S1. Construct a social media multimodal dataset: Obtain and download a public dataset containing image modality datasets and text modality datasets from public network resources as the original dataset. Standardize the image modality dataset and text modality dataset respectively, and use the BLIP-3 pre-trained model to extract deep features from the standardized image modality dataset and text modality dataset respectively to obtain deep image features and deep text features. Construct several samples based on the deep image features, deep text features and the original noise labels carried by the original dataset. The sample format is (deep image features, deep text features, original noise labels). Construct the final dataset containing training set, test set and retrieval set according to the original noise label category and the number of samples. S2. Constructing a Noise-Label Multimodal Hash Retrieval Model Based on Semantic Alignment Enhancement: Based on neural networks and deep learning techniques, a noise-label multimodal hash retrieval model based on semantic alignment enhancement is constructed, specifically: S201. Multimodal Fusion: Feature alignment is performed on deep image features and deep text features. The aligned deep image features and deep text features are concatenated and fused to generate a feature fusion matrix, and hash codes are obtained through a hash mapping network; S202. Noise-Aware Semantic Alignment Enhancement: The feature fusion matrix is ​​combined with the original noise labels. Reliable samples are dynamically identified through an adaptive label optimization process to generate clean soft pseudo-labels. Then, a category-level semantic prototype is constructed based on the clean soft pseudo-labels. Alignment optimization is performed on the hash code of each sample and the category-level semantic prototype corresponding to each sample's hash code in the hash space to obtain a noise-resistant robust hash code; S203. Hash Retrieval Query: The noise-resistant robust hash code is used for similarity retrieval through Hamming distance. Multiple iterations are performed, and the average precision is calculated. The average precision is used to evaluate the retrieval performance. S3. Training the model: Train the semantic alignment-enhanced noisy label multimodal hash retrieval model using the final dataset to obtain the trained semantic alignment-enhanced noisy label multimodal hash retrieval model.

[0024] As attached Figure 3 As shown, the public dataset in step S1 of this embodiment includes the MIRFlickr dataset, the NUS-WIDE dataset, and the MS COCO dataset; a set proportion of image-text pairs are randomly selected from the MIRFlickr dataset, the NUS-WIDE dataset, and the MS COCO dataset as the test set; the test set image-text pairs in the MIRFlickr dataset, the NUS-WIDE dataset, and the MS COCO dataset are excluded, and a portion is extracted from the remaining data as the retrieval set to ensure the diversity and representativeness of different diseases; then, in the retrieval dataset, the unselected image-text pairs are divided into the training set to ensure the richness of the training data.

[0025] The specific steps for obtaining depth image features in step S1 of this embodiment are as follows: S1-101. Preprocessing image modal data: The images in the image modal dataset are uniformly cropped and reshaped to an H×H pixel size, and the cropped and reshaped images are normalized so that the pixel values ​​of the cropped and reshaped images are in the range of [0,1], eliminating differences in brightness and contrast. S1-102. Extracting Depth Image Features: Depth image features are extracted using the BLIP-3 pre-trained model to capture subtle changes in the image and obtain depth image features for subsequent analysis.

[0026] For example, in the PyTorch framework, the key code is as follows: def preprocess_image(image_path, size=H): image = Image.open(image_path).convert('RGB') preprocess = transforms.Compose([ transforms.Resize((size, size)), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229,0.224, 0.225]) ]) image_tensor = preprocess(image) return image_tensor.unsqueeze(0) def extract_depth_features(image_tensor): processor = BlipProcessor.from_pretrained("Salesforce / blip-2") model = BlipForConditionalGeneration.from_pretrained("Salesforce / blip-2") inputs = processor(images=image_tensor, return_tensors="pt") with torch.no_grad(): features = model.get_image_features(**inputs) return features; Here, features represent the extracted depth image features.

[0027] The specific steps for obtaining deep text features in step S1 of this embodiment are as follows: S1-201. Preprocessing text modal data: Performing preprocessing operations such as text cleaning, text segmentation, and text vectorization on the text modal data, and mapping the text information into text vectors; S1-202. Extracting Deep Text Features: Deep text features are extracted using the BLIP-3 pre-trained model to capture detailed text description information, ensuring that the captured detailed text description information can reflect the complexity of the text content. The captured detailed text description information is then standardized to obtain deep text features, which are convenient for model learning and analysis.

[0028] For example, in the PyTorch framework, the key code is as follows: def preprocess_text(text): tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') tokens = tokenizer(text, padding=True, truncation=True, return_tensors='pt') return tokens['input_ids'], tokens['attention_mask'] def extract_text_features(input_ids, attention_mask): model = BertModel.from_pretrained('bert-base-uncased') with torch.no_grad(): outputs = model(input_ids=input_ids, attention_mask=attention_mask) features = outputs.last_hidden_state.mean(dim=1) return features; Here, features represent the extracted deep text features.

[0029] As attached Figure 4 As shown, the multimodal fusion in step S201 of this embodiment is as follows: S20101, Feature Alignment: Alignment depth image features and alignment depth text features are obtained through a feature aligner consisting of multiple linear layers and activation functions, as shown in the following formula: ; ; in, and These represent depth image features and depth text features, respectively. and These represent the image feature aligner and the text feature aligner, respectively. and Indicates training parameters; and These represent alignment depth image features and alignment depth text features, respectively. S20102, Feature Concatenation and Fusion: The aligned depth image features and aligned depth text features are processed by a feature concatenation and fusion function to generate a feature fusion matrix, as shown in the following formula: ; in, Indicates the feature splicing and fusion function; Represents the feature fusion matrix; and These represent alignment depth image features and alignment depth text features, respectively. Indicates a splicing operation; S20103, Hash Mapping: The feature fusion matrix is ​​processed by a hash mapping function to generate a relaxed hash code, and then processed... The function generates hash codes using the following formula: ; ; in, Represents a hash mapping function; Indicates the first The relaxed hash code of each sample; Indicates the first The hash code of each sample; Indicates the first Feature fusion matrix of each sample; This represents the training parameters.

[0030] For example, in the PyTorch framework, the key code is as follows: class FeatureAligner(nn.Module): def __init__(self, input_dim_x, input_dim_y, hidden_dim): super(FeatureAligner, self).__init__() self.linear_x = nn.Linear(input_dim_x, hidden_dim) self.linear_y = nn.Linear(input_dim_y, hidden_dim) self.activation = nn.ReLU() def forward(self, X, Y): F_x = self.activation(self.linear_x(X)) F_y = self.activation(self.linear_y(Y)) return F_x, F_y def integrate(F_x, F_y): return torch.cat((F_x, F_y), dim=1) class HashEncoder(nn.Module): def __init__(self, input_dim, output_dim): super(HashEncoder, self).__init__() self.linear = nn.Linear(input_dim, output_dim) def forward(self, H): b_i = self.linear(H) return b_i; Where nn.Linear represents a linear layer; FeatureAligner represents feature alignment; integrate represents feature concatenation and fusion; and HashEncoder represents a hash mapping function.

[0031] As attached Figure 5 As shown, the semantic alignment enhancement for noise perception in step S202 of this embodiment is as follows: S20201, Adaptive Label Optimization: The contrastive loss for each sample is calculated by combining the relaxed hash code and the original noisy label, as shown in the following formula: ; ; ; in, Indicates the first Contrast loss for each sample; Indicates the first The semantic center of a class; Indicates sample Category semantic confidence; Represents the contrast mask matrix; Indicates the original noise label; S20202. Generate adaptive clean soft pseudo-labels: Fit the contrastive loss of the samples to a bimodal Gaussian mixture model, calculate the posterior probability of a sample belonging to a clean label, and generate clean soft pseudo-labels based on an adaptive threshold, as shown in the following formula: ; ; in, Indicates the first The posterior probability of a sample; Indicates normalized contrast loss ; Indicates a clean, soft-sell label; Indicates an indicator function; Represents the mathematical expectation; Indicates variance; Indicates the number of samples; S20203, Prototype-guided semantic alignment enhancement: using clean soft pseudo-tags for categories Construct a semantic prototype and update it dynamically during training, as shown in the following formula: ; ; ; in, A positive class indicating a clean soft pseudo-label; Represents the semantic prototype; Cosine distance; Temperature coefficient; It is a numerically stable term; The number of samples with clean soft fake labels; Indicates the number of categories.

[0032] For example, in the PyTorch framework, the key code is as follows: def contrastive_loss(b_i, r_c, l_i, n): S_ic = b_i @ r_c.T M_ij = l_i.unsqueeze(1) * (1 - l_i.unsqueeze(0)) loss = 0 for j in range(n): for c in range(len(r_c)): loss += torch.norm(S_ic[:, c] - S_ic[j, c]) * M_ij[:, j] return loss / n def generate_soft_labels(contrastive_losses, n): model = GaussianMixture(n_components=2) model.fit(contrastive_losses.reshape(-1, 1)) prob = model.predict_proba(contrastive_losses.reshape(-1, 1)) posterior_probs = prob[:, 0] / (prob.sum(axis=1)) mean_prob = posterior_probs.mean() threshold = max(0.3, mean_prob) soft_labels = (posterior_probs>threshold).astype(int) return soft_labels class PrototypeAligner(nn.Module): def __init__(self, temperature=1.0, kappa=1e-5): super(PrototypeAligner, self).__init__() self.temperature = temperature self.kappa = kappa def forward(self, b_i, clean_labels): unique_labels = clean_labels.unique() prototypes = {} for label in unique_labels: if label == 0: continue prototypes[label.item()] = b_i[clean_labels == label].mean(dim=0) loss = 0 for idx, code in enumerate(b_i): label = clean_labels[idx].item() if label == 0: continue p_c = prototypes[label] d = torch.nn.functional.cosine_similarity(code.unsqueeze(0), p_c.unsqueeze(0)) loss -= torch.log(torch.exp(-d / self.temperature) +self.kappa) / (torch.exp(-d / self.temperature).sum() + self.kappa) return loss.mean(); Wherein, contrastive_loss represents S20201 adaptive label optimization; generate_soft_labels represents S20202 adaptive clean soft pseudo-label generation; and PrototypeAligner represents S20203 prototype-guided semantic alignment enhancement.

[0033] As attached Figure 6 As shown, the hash retrieval query in step S203 of this embodiment is as follows: S20301. Calculate the average accuracy using the following formula: ; in, This represents the total number of samples in the retrieval set that are associated with the query samples in the test set; This is the upper bound of the sample index; Indicates the relevant preceding The precision of the nth instance, when the retrieval set When a sample is relevant to a query sample in the test set ,otherwise ; S20302. Calculate the mean precision. By calculating the mean precision under different recall rates, the mean precision is averaged to obtain the MAP value, as shown in the following formula: ; in, Indicates the number of test sets; Indicates the first indivual value.

[0034] For example, in the PyTorch framework, the key code is as follows: def average_precision(y_true, y_scores): sorted_indices = torch.argsort(y_scores, descending=True) y_true_sorted = y_true[sorted_indices] N = y_true.sum().item() if N == 0: return 0.0 precision = [] for m in range(len(y_true_sorted)): if y_true_sorted[m] == 1: precision_m = (y_true_sorted[:m + 1].sum().item()) / (m +1) precision.append(precision_m) AP = sum(precision) / N return AP def mean_average_precision(y_true_list, y_scores_list): APs = [] for y_true, y_scores in zip(y_true_list, y_scores_list): AP = average_precision(y_true, y_scores) APs.append(AP) MAPQ = sum(APs) / len(APs) if APs else 0 return MAPQ; Where average_precision is the average precision (AP) calculated in S20301; mean_average_precision is the mean of average precision (MAP) calculated in S20302.

[0035] As attached Figure 7 As shown, the specific training model in step S3 of this embodiment is as follows: S301. Constructing the Loss Function: The goal of the semantically aligned and enhanced noisy label multimodal hash retrieval model is to utilize the feature fusion matrix and clean soft pseudo-labels, and to train an efficient multimodal hashing process through relaxed hash codes and hash codes, ultimately learning a semantically discriminative and noise-resistant multimodal hash retrieval system, utilizing the feature fusion matrix. Clean and soft fake labels Relaxed hash codes and hash code A multimodal hash retrieval model based on semantic alignment enhancement and noise label construction loss function is trained. S302. Model Optimization: The Adam algorithm is used as the optimization function for the semantic alignment-enhanced noisy label multimodal hash retrieval model. The model parameters are iteratively updated by minimizing the constructed loss function. The learning rate parameter is set to 0.001, and other hyperparameters use the default values ​​in PyTorch.

[0036] In this embodiment, the MAP value is selected as the evaluation index.

[0037] For example, the optimization function described above can be represented in PyTorch as follows: optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate) Here, torch.optim.Adam is the Adam optimization function embedded in PyTorch; model.parameters() is the set of parameters for model training; and learning_rate is the learning rate.

[0038] The specific construction of the loss function in step S301 of this embodiment is as follows: S30101. Construct the similarity preservation loss, the formula is as follows: ; ; in, , Represents a relaxed hash code; Indicates a clean, soft-sell label; Indicates cosine similarity; The number of samples with clean soft fake labels; Indicates the similarity of the reconstructions; S30102. Construct the quantization loss, the formula is as follows: ; in, Represents a relaxed hash code; Represents the hash code.

[0039] For example, the loss function described above can be represented in PyTorch as follows: def cosine_similarity(a, b): return torch.nn.functional.cosine_similarity(a.unsqueeze(1),b.unsqueeze(0), dim=2) def similarity_preserving_loss(b, soft_labels): t = b.size(0) similarity_matrix = cosine_similarity(b, b) T = (torch.sign(soft_labels.unsqueeze(1) @ soft_labels.unsqueeze(0)) * (1 - torch.exp(-soft_labels.unsqueeze(1) @ soft_labels.unsqueeze(0)))) loss = torch.sum((similarity_matrix - T) ** 2) return loss def quantization_loss(b, B): loss = torch.sum((B - b) ** 2) return loss; Wherein, similarity_preserving_loss represents the S30101 similarity preservation loss; quantization_loss represents the S30102 quantization loss.

[0040] The semantic alignment-enhanced noisy label multimodal hash retrieval model of this embodiment was tested on three public datasets: MIRFlickr, NUS-WIDE, and MS COCO. The experiments show that the model of this embodiment outperforms existing models. The comparison of the experimental results MAP is shown in Table 1.

[0041] Table 1. Comparison of MAP results from experiments

[0042] The model in this embodiment was compared with existing models, and it can be seen that the performance of this embodiment is the best compared with other methods.

[0043] Example 2: This embodiment provides a noise tag multimodal hash retrieval system based on semantic alignment enhancement. This system is used to implement the noise tag multimodal hash retrieval method based on semantic alignment enhancement as described in Embodiment 1. The system includes: The dataset construction unit is used to obtain and download public datasets containing image modality datasets and text modality datasets from public network resources as the original datasets. The image modality dataset and the text modality dataset are standardized respectively. Then, the BLIP-3 pre-trained model is used to extract deep features from the standardized image modality dataset and the text modality dataset to obtain deep image features and deep text features. Based on the deep image features, deep text features and the original noise labels carried by the original dataset, several samples are constructed. The sample format is (deep image features, deep text features, original noise labels). Based on the original noise label category and the number of samples, the final dataset containing training set, test set and retrieval set is constructed. The model building unit, based on neural networks and deep learning techniques, first aligns deep image features and deep text features. It then concatenates and fuses the aligned features to generate a feature fusion matrix, and obtains hash codes through a hash mapping network. Next, it combines the feature fusion matrix with the original noisy labels and dynamically identifies reliable samples through an adaptive label optimization process to generate clean soft pseudo-labels. Based on these clean soft pseudo-labels, it constructs category-level semantic prototypes and aligns and optimizes the hash codes of each sample and their corresponding category-level semantic prototypes in the hash space to obtain noise-resistant robust hash codes. Finally, it performs similarity retrieval using Hamming distance on these noise-resistant robust hash codes, iterating multiple times and calculating the average precision, which is used to evaluate retrieval performance. The model training module is used to train the semantic alignment-enhanced noisy label multimodal hash retrieval model using the final dataset, and obtain the trained semantic alignment-enhanced noisy label multimodal hash retrieval model.

[0044] As attached Figure 2 As shown, the model building unit in this embodiment includes: The feature alignment module is used to obtain aligned depth image features and aligned depth text features through a feature aligner consisting of multiple linear layers and activation functions, as shown in the following formula: ; ; in, and These represent depth image features and depth text features, respectively. and These represent the image feature aligner and the text feature aligner, respectively. and Indicates training parameters; and These represent alignment depth image features and alignment depth text features, respectively. The feature concatenation and fusion module is used to generate a feature fusion matrix by passing the alignment depth image features and alignment depth text features through a feature concatenation and fusion function, as shown in the following formula: ; in, Indicates the feature splicing and fusion function; Represents the feature fusion matrix; and These represent alignment depth image features and alignment depth text features, respectively. Indicates a splicing operation; The hash mapping module is used to generate relaxed hash codes from the feature fusion matrix through a hash mapping function, and then... The function generates hash codes using the following formula: ; ; in, Represents a hash mapping function; Indicates the first The relaxed hash code of each sample; Indicates the first The hash code of each sample; Indicates the first Feature fusion matrix of each sample; Indicates training parameters; The adaptive label optimization module combines the relaxed hash code and the original noisy label to calculate the contrastive loss for each sample, as shown in the following formula: ; ; ; in, Indicates the first Contrast loss for each sample; Indicates the first The semantic center of a class; Indicates sample Category semantic confidence; Represents the contrast mask matrix; Indicates the original noise label; The clean soft pseudo-label generation module is used to fit the contrastive loss of the samples to a bimodal Gaussian mixture model, calculate the posterior probability of a sample belonging to a clean label, and generate clean soft pseudo-labels based on an adaptive threshold, as shown in the following formula: ; ; in, Indicates the first The posterior probability of a sample; Indicates normalized contrast loss ; Indicates a clean, soft-sell label; Indicates an indicator function; Represents the mathematical expectation; Indicates variance; Indicates the number of samples; A prototype-guided semantic alignment enhancement module for using clean soft pseudo-labels for categories. Construct a semantic prototype and update it dynamically during training, as shown in the following formula: ; ; ; in, A positive class indicating a clean soft pseudo-label; Represents the semantic prototype; Cosine distance; Temperature coefficient; It is a numerically stable term; The number of samples with clean soft fake labels; Indicates the number of categories; The Average Precision (MAP) calculation module is used to calculate the mean precision under different recall rates, and then average the mean precision to obtain the MAP value. The formula is as follows: ; ; in, This represents the total number of samples in the retrieval set that are associated with the query samples in the test set; This is the upper bound of the sample index; Indicates the relevant preceding The precision of the nth instance, when the retrieval set When a sample is relevant to a query sample in the test set ,otherwise ; Indicates the number of test sets; Indicates the first indivual value.

[0045] Example 3: This embodiment also provides an electronic device, including: a memory and a processor; The memory stores the instructions executed by the computer. The processor executes computer execution instructions stored in the memory, causing the processor to perform the noise tag multimodal hash retrieval method based on semantic alignment enhancement in any embodiment of the present invention.

[0046] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0047] Memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0048] Example 4: This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the semantic alignment-enhanced noise tag multimodal hash retrieval method according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0049] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0050] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0051] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby achieving the function of any of the embodiments described above.

[0052] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A noise tag multimodal hash retrieval method based on semantic alignment enhancement, characterized in that, The method is as follows: Constructing a social media multimodal dataset: Obtain and download public datasets containing image and text modal datasets from public network resources as the original dataset. Standardize the image and text modal datasets respectively, and use the BLIP-3 pre-trained model to extract deep features from the standardized image and text modal datasets to obtain deep image features and deep text features. Construct several samples based on the deep image features, deep text features, and the original noise labels carried by the original dataset. The sample format is (deep image features, deep text features, original noise labels). Construct the final dataset containing training set, test set, and retrieval set according to the original noise label category and the number of samples. Constructing a Noise-Aligned Enhanced Multimodal Hash Retrieval Model: Based on neural networks and deep learning techniques, a noise-aligned multimodal hash retrieval model is constructed, specifically: Multimodal Fusion: Feature alignment is performed on deep image features and deep text features. The aligned deep image features and deep text features are then concatenated and fused to generate a feature fusion matrix, and hash codes are obtained through a hash mapping network; Noise-Aware Semantic Alignment Enhancement: The feature fusion matrix is ​​combined with the original noise labels. Reliable samples are dynamically identified through an adaptive label optimization process to generate clean soft pseudo-labels. Category-level semantic prototypes are then constructed based on these clean soft pseudo-labels. Alignment optimization is performed on the hash code of each sample and the corresponding category-level semantic prototype in the hash space to obtain noise-resistant robust hash codes. Hash retrieval query: The noise-resistant robust hash code is used for similarity retrieval through Hamming distance, and the process is iterated multiple times and the average precision is calculated. The average precision is used to evaluate the retrieval performance. Training the model: The semantic alignment-enhanced noisy label multimodal hash retrieval model is trained using the final dataset to obtain the trained semantic alignment-enhanced noisy label multimodal hash retrieval model.

2. The noise tag multimodal hash retrieval method based on semantic alignment enhancement according to claim 1, characterized in that, The specific steps for obtaining depth image features are as follows: Preprocessing image modal data: The images in the image modal dataset are uniformly cropped and reshaped to an H×H pixel size, and the cropped and reshaped images are normalized so that the pixel values ​​of the cropped and reshaped images are in the range of [0,1], eliminating differences in brightness and contrast; Extracting depth image features: The BLIP-3 pre-trained model is used to extract depth image features, capturing subtle changes in the image and thus obtaining depth image features; The specific steps for obtaining deep text features are as follows: Preprocessing text modal data: The text in the text modal data is preprocessed by text cleaning, text segmentation and text vectorization, and the text information is mapped into text vectors; Deep text feature extraction: Deep text feature extraction is performed using the BLIP-3 pre-trained model to capture detailed text description information, ensuring that the captured detailed text description information can reflect the complexity of the text content, and the captured detailed text description information is standardized to obtain deep text features.

3. The noise tag multimodal hash retrieval method based on semantic alignment enhancement according to claim 1 or 2, characterized in that, Multimodal fusion is detailed below: Feature alignment: Alignment depth image features and alignment depth text features are obtained through a feature aligner consisting of multiple linear layers and activation functions, as shown in the following formula: ; ; in, and These represent depth image features and depth text features, respectively. and These represent the image feature aligner and the text feature aligner, respectively. and Indicates training parameters; and These represent alignment depth image features and alignment depth text features, respectively. Feature concatenation and fusion: Alignment depth image features and alignment depth text features are processed by a feature concatenation and fusion function to generate a feature fusion matrix, as shown in the following formula: ; in, Indicates the feature splicing and fusion function; Represents the feature fusion matrix; and These represent alignment depth image features and alignment depth text features, respectively. Indicates a splicing operation; Hash mapping: The feature fusion matrix is ​​processed by a hash mapping function to generate a relaxed hash code, and then processed... The function generates hash codes using the following formula: ; ; in, Represents a hash mapping function; Indicates the first The relaxed hash code of each sample; Indicates the first The hash code of each sample; Indicates the first Feature fusion matrix of each sample; This represents the training parameters.

4. The noise tag multimodal hash retrieval method based on semantic alignment enhancement according to claim 3, characterized in that, The specific details of noise-aware semantic alignment enhancement are as follows: Adaptive label optimization: The contrastive loss for each sample is calculated by combining the relaxed hash code and the original noisy label, as shown in the following formula: ; ; ; in, Indicates the first Contrast loss for each sample; Indicates the first The semantic center of a class; Indicates sample Category semantic confidence; Represents the contrast mask matrix; Indicates the original noise label; Generate adaptive clean soft pseudo-labels: Fit the contrastive loss of the samples to a bimodal Gaussian mixture model, calculate the posterior probability of a sample belonging to a clean label, and generate clean soft pseudo-labels based on an adaptive threshold, as shown in the following formula: ; ; in, Indicates the first The posterior probability of a sample; Indicates normalized contrast loss ; Indicates a clean, soft-sell label; Indicates an indicator function; Represents the mathematical expectation; Indicates variance; Indicates the number of samples; Prototype-guided semantic alignment enhancement: leveraging clean soft pseudo-tags for categories Construct a semantic prototype and update it dynamically during training, as shown in the following formula: ; ; ; in, A positive class indicating a clean soft pseudo-label; Represents the semantic prototype; Cosine distance; Temperature coefficient; It is a numerically stable term; The number of samples with clean soft fake labels; Indicates the number of categories.

5. The noise tag multimodal hash retrieval method based on semantic alignment enhancement according to claim 4, characterized in that, The hash search query is as follows: The average accuracy is calculated using the following formula: ; in, This represents the total number of samples in the retrieval set that are associated with the query samples in the test set; This is the upper bound of the sample index; Indicates the relevant preceding The precision of the nth instance, when the retrieval set When a sample is relevant to a query sample in the test set ,otherwise ; The mean precision (MAP) is calculated by averaging the mean precision at different recall rates. The formula is as follows: ; in, Indicates the number of test sets; Indicates the first indivual value.

6. The noise tag multimodal hash retrieval method based on semantic alignment enhancement according to claim 5, characterized in that, The specific training model is as follows: Constructing the loss function: The goal of the semantically aligned and enhanced noisy label multimodal hash retrieval model is to utilize the feature fusion matrix and clean soft pseudo-labels, and to train an efficient multimodal hashing process through relaxed hash codes and hash codes, ultimately learning a semantically discriminative and noise-resistant multimodal hash retrieval system, utilizing the feature fusion matrix. Clean and soft fake labels Relaxed hash codes and hash code A loss function is constructed to train a noisy label multimodal hash retrieval model based on semantic alignment enhancement; the details are as follows: The similarity preservation loss is constructed as follows: ; ; in, , Represents a relaxed hash code; Indicates a clean, soft-sell label; Indicates cosine similarity; The number of samples with clean soft fake labels; Indicates the similarity of the reconstructions; The quantization loss is constructed using the following formula: ; in, Represents a relaxed hash code; Represents the hash code; Model optimization: The Adam algorithm is used as the optimization function for the semantic alignment-enhanced noisy label multimodal hash retrieval model. The model parameters are iteratively updated by minimizing the constructed loss function.

7. A noisy tag multimodal hash retrieval system based on semantic alignment enhancement, characterized in that, This system is used to implement the noise tag multimodal hash retrieval method based on semantic alignment enhancement as described in any one of claims 1 to 6; the system comprises: The dataset construction unit is used to obtain and download public datasets containing image modality datasets and text modality datasets from public network resources as the original datasets. The image modality dataset and the text modality dataset are standardized respectively. Then, the BLIP-3 pre-trained model is used to extract deep features from the standardized image modality dataset and the text modality dataset to obtain deep image features and deep text features. Based on the deep image features, deep text features and the original noise labels carried by the original dataset, several samples are constructed. The sample format is (deep image features, deep text features, original noise labels). Based on the original noise label category and the number of samples, the final dataset containing training set, test set and retrieval set is constructed. The model building unit, based on neural networks and deep learning techniques, first aligns deep image features and deep text features. It then concatenates and fuses the aligned features to generate a feature fusion matrix, and obtains hash codes through a hash mapping network. Next, it combines the feature fusion matrix with the original noisy labels and dynamically identifies reliable samples through an adaptive label optimization process to generate clean soft pseudo-labels. Based on these clean soft pseudo-labels, it constructs category-level semantic prototypes and aligns and optimizes the hash codes of each sample and their corresponding category-level semantic prototypes in the hash space to obtain noise-resistant robust hash codes. Finally, it performs similarity retrieval using Hamming distance on these noise-resistant robust hash codes, iterating multiple times and calculating the average precision, which is used to evaluate retrieval performance. The model training module is used to train the semantic alignment-enhanced noisy label multimodal hash retrieval model using the final dataset, and obtain the trained semantic alignment-enhanced noisy label multimodal hash retrieval model.

8. The noise tag multimodal hash retrieval system based on semantic alignment enhancement according to claim 7, characterized in that, The model building units include: The feature alignment module is used to obtain aligned depth image features and aligned depth text features through a feature aligner consisting of multiple linear layers and activation functions, as shown in the following formula: ; ; in, and These represent depth image features and depth text features, respectively. and These represent the image feature aligner and the text feature aligner, respectively. and Indicates training parameters; and These represent alignment depth image features and alignment depth text features, respectively. The feature concatenation and fusion module is used to generate a feature fusion matrix by passing the alignment depth image features and alignment depth text features through a feature concatenation and fusion function, as shown in the following formula: ; in, Indicates the feature splicing and fusion function; Represents the feature fusion matrix; and These represent alignment depth image features and alignment depth text features, respectively. Indicates a splicing operation; The hash mapping module is used to generate relaxed hash codes from the feature fusion matrix through a hash mapping function, and then... The function generates hash codes using the following formula: ; ; in, Represents a hash mapping function; Indicates the first The relaxed hash code of each sample; Indicates the first The hash code of each sample; Indicates the first Feature fusion matrix of each sample; Indicates training parameters; The adaptive label optimization module combines the relaxed hash code and the original noisy label to calculate the contrastive loss for each sample, as shown in the following formula: ; ; ; in, Indicates the first Contrast loss for each sample; Indicates the first The semantic center of a class; Indicates sample Category semantic confidence; Represents the contrast mask matrix; Indicates the original noise label; The clean soft pseudo-label generation module is used to fit the contrastive loss of the samples to a bimodal Gaussian mixture model, calculate the posterior probability of a sample belonging to a clean label, and generate clean soft pseudo-labels based on an adaptive threshold, as shown in the following formula: ; ; in, Indicates the first The posterior probability of a sample; Indicates normalized contrast loss ; Indicates a clean, soft-sell label; Indicates an indicator function; Represents the mathematical expectation; Indicates variance; Indicates the number of samples; A prototype-guided semantic alignment enhancement module for using clean soft pseudo-labels for categories. Construct a semantic prototype and update it dynamically during training, as shown in the following formula: ; ; ; in, A positive class indicating a clean soft pseudo-label; Represents the semantic prototype; Cosine distance; Temperature coefficient; It is a numerically stable term; The number of samples with clean soft fake labels; Indicates the number of categories; The Average Precision (MAP) calculation module is used to calculate the mean precision under different recall rates, and then average the mean precision to obtain the MAP value. The formula is as follows: ; ; in, This represents the total number of samples in the retrieval set that are associated with the query samples in the test set; This is the upper bound of the sample index; Indicates the relevant preceding The precision of the nth instance, when the retrieval set When a sample is relevant to a query sample in the test set ,otherwise ; Indicates the number of test sets; Indicates the first indivual value.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the noise tag multimodal hash retrieval method based on semantic alignment enhancement as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the semantic alignment-enhanced noise tag multimodal hash retrieval method as described in any one of claims 1 to 6.

Images