A cross-modal weighted hashing retrieval method oriented to semantic alignment

Abstract

A cross-modal weighted hash retrieval method for semantic alignment includes the following steps: Step 1: Accurate extraction and semantic alignment of image and text features are achieved through a dynamic masking mechanism; Step 2: High-quality binary hash codes are generated through channel adaptive weighting and contrastive loss optimization; Step 3: Cross-modal hash matching retrieval is performed. The purpose of this invention is to effectively narrow the cross-modal semantic gap, achieve accurate semantic alignment, and simultaneously consider the differences in feature importance and model generalization ability, thereby improving the quality and retrieval performance of existing cross-modal hash retrieval technologies. This invention provides a cross-modal retrieval method based on dynamic masking and assisted hash learning.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more particularly to big data and artificial intelligence technologies, specifically to a cross-modal weighted hash retrieval method oriented towards semantic alignment. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, the generation and application of multimodal data (images, text, audio, video, etc.) are experiencing explosive growth. Cross-modal retrieval, as a key technology connecting different types of data, enables cross-type information matching such as "searching for images by text" and "searching for text by images," and has irreplaceable application value in fields such as intelligent retrieval, content recommendation, digital twins, and medical image analysis.

[0003] Traditional unimodal retrieval techniques can only handle data of the same type and cannot meet the retrieval needs of multimodal scenarios. Cross-modal hash retrieval, with its advantages of high efficiency and low storage cost, has become a research hotspot in this field. The core of this technology is to map high-dimensional multimodal data to a low-dimensional binary hash code space and achieve fast matching by calculating the Hamming distance between hash codes. In recent years, the combination of deep learning technology and cross-modal hash retrieval has made significant progress. Convolutional neural networks (CNNs) and generative adversarial networks (GANs) have been used to extract deep semantic features of images, while large language models (LLMs) have enhanced the semantic representation capabilities of text modalities, further improving the accuracy of cross-modal matching.

[0004] However, existing cross-modal hash retrieval technologies still face many challenges. The semantic representations of data from different modalities naturally differ, and effectively bridging the cross-modal semantic gap and achieving accurate alignment of features from different modalities is a core problem that urgently needs to be solved in this field. At the same time, noise interference and imbalanced channel information weights during feature extraction also seriously affect the quality of hash codes and retrieval performance.

[0005] The existing technology has the following technical defects: 1) The cross-modal semantic gap is difficult to bridge: The distribution of underlying features differs significantly across different modalities. For example, image data is dominated by visual features such as color, texture, and shape, while text data is dominated by semantic features such as vocabulary, grammar, and context. Most existing methods employ simple feature fusion strategies, which fail to fully explore the deep semantic relationships between different modalities. This results in large hash code distances between similar samples, making matching errors more likely and hindering the improvement of retrieval accuracy. 2) Imbalanced channel information weights lead to insufficient capture of key features: During feature extraction, the contributions of features from different channels to the retrieval task vary. Traditional hash layers use a uniform weighting method, ignoring the differences in importance between channels and failing to effectively filter key feature channels. A large amount of redundant information and noise is retained, which not only increases computational complexity but also reduces the discriminative power of hash codes, resulting in poor robustness of retrieval results. 3) The single hash code optimization objective limits the distinguishing ability: Existing methods usually only minimize the hash code distance between similar samples as the single optimization objective, lacking a constraint mechanism for dissimilar samples. This results in a relatively dense distribution of generated hash codes in high-dimensional space, with dissimilar samples having excessively close hash code distances, making them difficult to distinguish effectively, thus affecting the accuracy and stability of cross-modal retrieval; 4) Insufficient model generalization ability: Some existing methods rely on the distribution characteristics of specific datasets, resulting in significant performance degradation in cross-dataset scenarios. Furthermore, most of these methods are designed for image-text modalities, making it difficult to extend to more modal types such as audio and video, thus limiting their applicability and failing to meet the needs of complex multimodal retrieval scenarios.

[0006] Therefore, the applicant proposes a cross-modal weighted hash retrieval method oriented towards semantic alignment. Summary of the Invention

[0007] The purpose of this invention is to effectively narrow the cross-modal semantic gap and achieve accurate semantic alignment, while taking into account the differences in feature importance and model generalization ability, and to improve the quality and retrieval performance of existing cross-modal hash retrieval technologies. This invention provides a cross-modal retrieval method based on dynamic masking and assisted hash learning. This method uses a dynamic masking mechanism to select key semantic features, an assisted hash layer to achieve adaptive adjustment of channel weights, and a comparative loss function to optimize hash code distribution, ultimately achieving a dual improvement in cross-modal retrieval accuracy and robustness.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A cross-modal weighted hash retrieval method oriented towards semantic alignment includes the following steps: Step 1: Achieve accurate extraction and semantic alignment of image and text features through a dynamic masking mechanism; Step 2: Generate high-quality binary hash codes through channel adaptive weighting and contrastive loss optimization; Step 3: Perform cross-modal hash matching retrieval.

[0009] Step 1 includes the following steps: Step 1-1) Learn image features; specifically including: Sub-step 1-1-1) Perform image segmentation and linear mapping; Sub-step 1-1-2) adds the location code; Sub-step 1-1-3) performs feature filtering for the dynamic masking mechanism; Steps 1-2) involve learning text features; specifically including: Sub-step 1-2-1) Encodes text sub-words; Sub-step 1-2-2) performs position encoding and dynamic masking processing; Steps 1-3) The image features obtained in step 1-1) and the text features obtained in step 1-2) are mapped to a unified semantic feature space after being processed by the same encoder and dynamic mask.

[0010] In sub-step 1-1-1), the input dimension is... The image data is divided into segments of size [size missing]. Calculate the total number of non-overlapping image patches. Where H is the image height, W is the image width, C is the number of image channels, and P is the image block size; Flatten each image patch into a one-dimensional vector to construct the image patch feature matrix. ; Through linear transformation matrix Map image patch features to a dimension of The semantic feature space is given by the formula: ; in, For the bias term, the mapped features It preserves both the low-level visual features and high-level semantic features of the image.

[0011] In sub-step 1-1-2), to preserve the spatial location information of the image patch, a learnable location code is added to the mapped features. This yields image features with location information: ; The introduction of location encoding enables the model to distinguish image patches at different locations, improving the spatial resolution of features.

[0012] In sub-step 1-1-3), to address the issue of overly even weight distribution in traditional attention mechanisms, a dynamic mask matrix is ​​introduced. To filter image features, the mask matrix is ​​defined as follows: ; when When the model normally calculates the attention weights at that location, it does so; when... At that time, the attention score at that location is completely masked, and the model focuses only on the key image patch features; The position-encoded features are input into the Transformer encoder, and multi-head self-attention is calculated using a dynamic mask matrix, as shown in the formula: ; in, , , These are query, key, and value characteristics, respectively. For key feature dimension, M is the corresponding transformation matrix; M is the mask matrix. This is the transpose of the Key vector; Finally, the image semantic features are obtained through processing using a feedforward neural network MLP. ,in, Let i be the i-th input feature vector. To The output after MLP transformation.

[0013] In sub-step 1-2-1), the specific steps are as follows: Input text data and perform sub-word segmentation using byte pair encoding (BPE); this method effectively solves the problem of out-of-vocabulary words for rare words by statistically analyzing the frequency of character pairs in the text and merging high-frequency character pairs into new sub-word units; after segmentation, a sub-word unit of length is generated. The subword sequence is mapped to a dimension of through the embedding layer. Text feature matrix .

[0014] In sub-step 1-2-2), consistent with the image feature learning process, the specific steps are as follows: add positional encoding to the text features. ,get To preserve the word order information of the text, the positionally encoded text features are input into the Transformer encoder, which uses the same dynamic masking mechanism as image features to filter key semantic features (such as core nouns and verbs) in the text, and finally outputs the text semantic features. ,in, For the multilayer perceptron to act on the input .

[0015] Step 2 generates high-quality binary hash codes through channel adaptive weighting and contrastive loss optimization, solving the problems of imbalanced channel weights and singular optimization objectives in traditional hash layers. Specifically, it includes the following steps: Step 2-1) Perform linear dimensionality reduction and nonlinear activation of the features; Step 2-2) Perform adaptive channel weighting: To address the imbalance of channel information weights, design a channel weighting unit to adaptively calculate the importance weight of each channel; Steps 2-3) generate the hash code; In step 2-1), the specific procedure is as follows: Image features and text features The input is a linear dimensionality reduction layer, mapped to a low-dimensional embedding space, reducing computational complexity. The formula is as follows: ; in, For a dimension reduction matrix, Let D×d be a real matrix. For bias terms, Let d be a real vector. The feature dimension after dimensionality reduction; The ReLU activation function is used to perform a nonlinear transformation on the dimensionality-reduced features, thereby enhancing the expressive power of the features: ; in, Nonlinear and sparsity processing of image features This involves non-linear and sparsity processing of text features.

[0016] Step 2-2) includes the following steps: Sub-step 2-2-1) Calculate global channel statistics: Features after activation and Perform global average pooling to obtain statistical information for each channel. : ; in, Representing image (I) or text (T) modality, The spatial dimension of the feature For the number of channels, It is a real vector of dimension C; Sub-step 2-2-2) Calculate channel weights: Statistical information Two fully connected layers are input to achieve channel-level compression and restoration, capturing the non-linear correlation between channels; the first fully connected layer... Change the number of channels from Compress to , For compression ratio, channel-weighted intermediate features Activated via ReLU: ; in, This is the weight matrix of the first fully connected layer. This is a bias term used to adjust the activation threshold of neurons; This indicates that a nonlinear activation operation is applied to the result of the linear transformation; Second fully connected layer Restore the number of channels to The channel weight vector is obtained. : ; in, , ; The weights are normalized using the Sigmoid function. Interval: ; Sub-step 2-2-3) generates channel-weighted features: Weight vector With features , Channel-by-channel dot product enhances key channel features and suppresses redundant channel features: ; in, This indicates element-wise multiplication. and The final embedded features are the weighted values. In steps 2-3), the specific procedure is as follows: the weighted features are input into the linear transformation layer and mapped to the hash probability space. ; in, Let be the transformation matrix. For bias terms, Representing an image or text modality; calculating the probability of each hash bit being 0 or 1 using the softmax function: ; Generate binary hash codes using the maximum probability method: ; Among them B i This represents the binary hash code of the i-th bit, and the final generated hash code is of length . Hash vector: ; in, This refers to the binary hash code of the k-th candidate sample.

[0017] Step 2 also includes steps 2-4) to optimize the contrastive loss function: To improve the discriminative power of hash codes, maximize the similarity between positive sample pairs (images and text), and minimize the similarity between negative samples, a contrastive loss function is designed to constrain the hash code distance between samples of the same class and samples of different classes. 2-4-1) Feature normalization: First, the mapped image features f i and text features f t Let X1 and X2 be represented respectively; L2 normalization is performed on the image and text embedding features to avoid similarity calculation bias caused by differences in feature vector lengths: ; in, This represents the global feature vector of the image after encoder and nonlinear transformation, used for subsequent hash mapping or cross-modal matching; This represents the global feature vector of the text after encoder and nonlinear transformation, used for cross-modal matching with image features; 2-4-2) Similarity calculation: Calculate the cosine similarity matrix between normalized features to measure the degree of matching between samples across modalities: ; , This represents the feature representation of the i-th image patch after nonlinear transformation or attention weighting, which is used for subsequent interaction with text features. This represents the feature representation of the j-th text word after nonlinear transformation or attention weighting, used for interaction with image features; 2-4-3) Loss Function Construction: Positive sample loss is defined as minimizing the similarity distance between similar samples (matching image-text pairs), and the formula is: ; in, For the label matrix, Indicates the first The image and the first Each text is a matching pair; The negative sample loss is defined as maximizing the similarity distance between out-of-class samples (mismatched image-text pairs), and the formula is: ; in, This is a similarity threshold used to control the minimum distance between dissimilar samples; The total loss is a weighted sum of the positive and negative sample losses, which serves as the optimization objective of the model. ; By minimizing the total loss, the hash codes generated by the model can achieve the distribution characteristics of samples of the same type being close in distance and samples of different types being far in distance, which significantly improves the accuracy of cross-modal retrieval.

[0018] Compared with the prior art, the present invention has the following technical effects: 1) Precisely filtering key semantic information to significantly narrow the cross-modal semantic gap: Traditional cross-modal retrieval methods employ a global attention mechanism, assigning average weights to image blocks and text sub-words. This is susceptible to background noise and redundant information, resulting in low semantic alignment accuracy between different modalities. The dynamic masking mechanism introduced in this invention can adaptively generate a mask matrix based on the length of the input sequence, masking feature information at non-critical locations and forcing the model to focus on the core regions of the image and the core words of the text. Simultaneously, image and text features share the same Transformer encoder and masking strategy, further strengthening the consistency of cross-modal semantics and solving the core problem of "semantic heterogeneity" in data from different modalities. 2) Adaptive balancing of channel weights to enhance the semantic expressiveness of hash codes: Traditional hash layers use a uniform processing method for all feature channels, ignoring the differences in contribution of different channels to the retrieval task. This results in key features being masked by redundant channels, leading to insufficient semantic recognizability of hash codes. The auxiliary hash layer designed in this invention captures global statistical information of each channel through global average pooling, and then adaptively calculates the channel importance weights through a fully connected layer structure of "channel compression-recovery". This process can accurately identify and strengthen the features of key channels, while suppressing the interference of redundant channels, thereby increasing the information entropy of hash codes by more than 20%. 3) Enhanced Discrimination of Hash Codes Through Dual Constraints in the Contrast Loss Function: Existing methods often employ a single optimization objective, focusing only on reducing the hash code distance between similar samples, lacking constraints on dissimilar samples. This leads to densely distributed hash codes in high-dimensional space, easily causing "dissimilar sample confusion" during retrieval. The contrast loss function constructed in this invention achieves the optimization objective of "bringing the distance between similar samples and pushing the distance between dissimilar samples" through dual constraints of positive sample loss and negative sample loss. The positive sample loss forces matching of image-text pairs with highly similar hash codes, while the negative sample loss uses a similarity threshold. Ensure that the hash code distance of mismatched samples is greater than a set threshold; 4) Excellent scalability and versatility, supporting multimodal data retrieval: The dynamic masking, channel weighting, and contrast loss of this invention are universally applicable, not only suitable for image-text cross-modal retrieval but also extend to retrieval tasks of more modal types such as audio and video. Only targeted adjustments to the feature extraction submodule are needed, such as using an audio Transformer to extract audio features and a video frame feature fusion strategy to extract video features, to quickly adapt to new modal combinations, solving the problem of poor modal adaptability in traditional methods. Attached Figure Description

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments: Figure 1 This is a schematic diagram of the overall process of the present invention; Figure 2 This is a flowchart of the dynamic masking mechanism in this invention. Detailed Implementation

[0020] like Figure 1 As shown, a cross-modal weighted hash retrieval method oriented towards semantic alignment includes the following steps: Step 1: Achieve accurate extraction and semantic alignment of image and text features through a dynamic masking mechanism; Step 2: Generate high-quality binary hash codes through channel adaptive weighting and contrastive loss optimization; Step 3: Perform cross-modal hash matching retrieval.

[0021] Step 1 includes the following steps: Step 1-1) Learn image features; specifically including: Sub-step 1-1-1) Perform image segmentation and linear mapping; Sub-step 1-1-2) adds the location code; Sub-step 1-1-3) performs feature filtering for the dynamic masking mechanism; Steps 1-2) involve learning text features; specifically including: Sub-step 1-2-1) Encodes text sub-words; Sub-step 1-2-2) performs position encoding and dynamic masking processing; Steps 1-3) The image features obtained in step 1-1) and the text features obtained in step 1-2) are mapped to a unified semantic feature space after being processed by the same encoder and dynamic mask.

[0022] In sub-step 1-1-1), the input dimension is... The image data is divided into segments of size [size missing]. Calculate the total number of non-overlapping image patches. Where H is the image height, W is the image width, C is the number of image channels, and P is the image block size; Flatten each image patch into a one-dimensional vector to construct the image patch feature matrix. ; Through linear transformation matrix Map image patch features to a dimension of The semantic feature space is given by the formula: ; in, For the bias term, the mapped features It preserves both the low-level visual features and high-level semantic features of the image.

[0023] In sub-step 1-1-2), to preserve the spatial location information of the image patch, a learnable location code is added to the mapped features. This yields image features with location information: ; The introduction of location encoding enables the model to distinguish image patches at different locations, improving the spatial resolution of features.

[0024] In sub-step 1-1-3), to address the issue of overly even weight distribution in traditional attention mechanisms, a dynamic mask matrix is ​​introduced. To filter image features, the mask matrix is ​​defined as follows: ; when When the model normally calculates the attention weights at that location, it does so; when... At that time, the attention score at that location is completely masked, and the model focuses only on the key image patch features; The position-encoded features are input into the Transformer encoder, and multi-head self-attention is calculated using a dynamic mask matrix, as shown in the formula: ; in, , , These are query, key, and value characteristics, respectively. For key feature dimension, M is the corresponding transformation matrix; M is the mask matrix. This is the transpose of the Key vector; Finally, the image semantic features are obtained through processing using a feedforward neural network MLP. ,in, Let i be the i-th input feature vector. To The output after MLP transformation.

[0025] In sub-step 1-2-1), the specific steps are as follows: Input text data and perform sub-word segmentation using byte pair encoding (BPE); this method effectively solves the problem of out-of-vocabulary words for rare words by statistically analyzing the frequency of character pairs in the text and merging high-frequency character pairs into new sub-word units; after segmentation, a sub-word unit of length is generated. The subword sequence is mapped to a dimension of through the embedding layer. Text feature matrix .

[0026] In sub-step 1-2-2), consistent with the image feature learning process, the specific steps are as follows: add positional encoding to the text features. ,get To preserve the word order information of the text, the positionally encoded text features are input into the Transformer encoder, which uses the same dynamic masking mechanism as image features to filter key semantic features (such as core nouns and verbs) in the text, and finally outputs the text semantic features. ,in, For the multilayer perceptron to act on the input .

[0027] Step 2 includes the following steps: Step 2-1) Perform linear dimensionality reduction and nonlinear activation of the features; Step 2-2) Perform adaptive weighting of the channels; Steps 2-3) generate the hash code; In step 2-1), the specific procedure is as follows: Image features and text features The input is a linear dimensionality reduction layer, mapped to a low-dimensional embedding space, reducing computational complexity. The formula is as follows: ; in, For a dimension reduction matrix, Let D×d be a real matrix. For bias terms, Let d be a real vector. The feature dimension after dimensionality reduction; The ReLU activation function is used to perform a nonlinear transformation on the dimensionality-reduced features, thereby enhancing the expressive power of the features: ; in, Nonlinear and sparsity processing of image features This involves non-linear and sparsity processing of text features.

[0028] Step 2-2) includes the following steps: Sub-step 2-2-1) Calculate global channel statistics: Features after activation and Perform global average pooling to obtain statistical information for each channel. : ; in, Representing image (I) or text (T) modality, The spatial dimension of the feature For the number of channels, It is a real vector of dimension C; Sub-step 2-2-2) Calculate channel weights: Statistical information Two fully connected layers are input to achieve channel-level compression and restoration, capturing the non-linear correlation between channels; the first fully connected layer... Change the number of channels from Compress to , For compression ratio, channel-weighted intermediate features Activated via ReLU: ; in, This is the weight matrix of the first fully connected layer. This is a bias term used to adjust the activation threshold of neurons; This indicates that a nonlinear activation operation is applied to the result of the linear transformation; Second fully connected layer Restore the number of channels to The channel weight vector is obtained. : ; in, , ; The weights are normalized using the Sigmoid function. Interval: ; Sub-step 2-2-3) generates channel-weighted features: Weight vector With features , Channel-by-channel dot product enhances key channel features and suppresses redundant channel features: ; in, This indicates element-wise multiplication. and The final embedded features are the weighted values. In steps 2-3), the specific procedure is as follows: the weighted features are input into the linear transformation layer and mapped to the hash probability space. ; in, Let be the transformation matrix. For bias terms, Representing an image or text modality; calculating the probability of each hash bit being 0 or 1 using the softmax function: ; Generate binary hash codes using the maximum probability method: ; Among them B i This represents the binary hash code of the i-th bit, and the final generated hash code is of length . Hash vector: .

[0029] Step 2 also includes steps 2-4) to optimize the contrastive loss function: To improve the discriminative power of hash codes, maximize the similarity between positive sample pairs (images and text), and minimize the similarity between negative samples, a contrastive loss function is designed to constrain the hash code distance between samples of the same class and samples of different classes. 2-4-1) Feature normalization: First, the mapped image features f i and text features f t Let X1 and X2 be represented respectively; L2 normalization is performed on the image and text embedding features to avoid similarity calculation bias caused by differences in feature vector lengths: ; 2-4-2) Similarity calculation: Calculate the cosine similarity matrix between normalized features to measure the degree of matching between samples across modalities: ; 2-4-3) Loss Function Construction: Positive sample loss is defined as minimizing the similarity distance between similar samples (matching image-text pairs), and the formula is: ; in, For the label matrix, Indicates the first The image and the first Each text is a matching pair; The negative sample loss is defined as maximizing the similarity distance between out-of-class samples (mismatched image-text pairs), and the formula is: ; in, This is a similarity threshold used to control the minimum distance between dissimilar samples; The total loss is a weighted sum of the positive and negative sample losses, which serves as the optimization objective of the model. ; By minimizing the total loss, the hash codes generated by the model can achieve the distribution characteristics of samples of the same type being close in distance and samples of different types being far in distance, which significantly improves the accuracy of cross-modal retrieval.

[0030] Example: 1. Implementation environment configuration: (1) Hardware environment; The method of this invention is implemented based on a high-performance computing server, with the specific hardware configuration as follows: Processor: Intel Xeon Gold 6330 2.0GHz 28-core; Graphics card: NVIDIA GeForce RTX 3080 GPU, 40GB VRAM; Memory: 256GB DDR4 ECC; Storage: 2TB NVMe SSD for storing datasets and model weights; Network: 10Gbps Ethernet, used for dataset download and model deployment; (2) Software environment; The method of this invention is developed based on the Python deep learning framework, and the specific software configuration is as follows: Operating system: Ubuntu 20.04 LTS 64-bit; Deep learning framework: PyTorch 2.0.1; Auxiliary libraries: CUDA 11.7, cuDNN 8.5.0, OpenCV 4.7.0, Transformers 4.30.2; Programming language: Python 3.9.16; Development tools: PyCharm 2023.2, Jupyter Notebook 6.5.4; 2. Specific implementation steps: (1) Operation flow of the cross-modal feature learning module; This embodiment details the process of extracting image and text features and filtering key information. The specific steps are as follows: Image feature extraction steps: ① Input a color image with dimensions of 224×224×3, divide the image into blocks of 16×16 pixels, and calculate the number of image blocks: ; ② Flatten each 16×16×3 image patch into a 768-dimensional vector to construct a 196×768 image patch feature matrix. ; ③ Through linear transformation matrix Mapping image patch features to a 768-dimensional semantic space, the formula is as follows: ,in , ; ④ Add learnable location codes To obtain features with location information ; ⑤ Generate a dynamic mask matrix For a sequence of 196 image patches, when hour, To block subsequent location information; when hour, Normal calculation of attention weights; ⑥ Input the Transformer encoder and compute multi-head self-attention: ;in, , , ; ⑦ The attention output is processed by a feedforward neural network (MLP) to obtain the final image semantic features. It has 768 dimensions.

[0031] Text feature extraction steps: (1) Input text and segment it into a word sequence using BPE encoding; (2) The word sequence is mapped into a 7×768 text feature matrix through the embedding layer. ; (3) Add location encoding ,get ; (4) Generate dynamic mask matrix For a sequence of 7 words, when hour, To block subsequent location information; when hour, ; (5) Input the Transformer encoder and perform the same attention calculation and MLP processing as for image features to obtain the final text semantic features. It has 768 dimensions.

[0032] (2) The operation flow of the auxiliary hash learning module; This embodiment details the process of feature dimensionality reduction, channel weighting, and hash code generation. The specific steps are as follows: Feature linear dimensionality reduction and activation: ① Image features and text features Input linear dimensionality reduction layer, mapped to a 256-dimensional embedding space: , ,in , ; ②Nonlinear transformation using the ReLU activation function: , ; Channel adaptive weighting: ① For 256-dimensional features and Perform global average pooling to calculate channel statistics: ,in The sequence length is 196 for images and 7 for text. ; ② Through Layer compression channel dimension: ,in , Compression ratio ; ③ Through Layer recovery channel dimension: ,in , ; ④ Generate channel weights using the Sigmoid function: , ; ⑤ Channel-by-channel weighted features: , Strengthen the characteristics of key channels.

[0033] Hash code generation: ①Weighted features and The input linear transformation layer is mapped to the hash probability space: ,in , , The hash code length (taken as 64); ② Calculate the probability distribution using the softmax function: ; ③ Generate binary hash codes using the maximum probability method: If ,but ,otherwise This ultimately generates a 64-bit hash vector. .

[0034] (3) Compare loss function optimization and model training; This embodiment details the model training process and the calculation logic of the loss function. The specific steps are as follows: Feature normalization: Image embedding features and text embedding features Perform L2 normalization to avoid the influence of vector length on similarity calculation: , ; Similarity matrix calculation: Calculate the cosine similarity of image-text feature pairs: Construct a similarity matrix.

[0035] Loss function calculation: ① Construct a label matrix If the first The image and the first If a text match, then ,otherwise ; ② Calculate the positive sample loss: Minimize the similarity distance between positive sample pairs; ③ Calculate the negative sample loss: The similarity of negative sample pairs is controlled to be below the threshold of 1. ④ Total loss: .

[0036] Model training and optimization: ① Minimize the total loss using the Adam optimizer The initial learning rate was set to 1e-3, and the learning rate decayed to 0.9 times every 5 epochs. ② Set the training batch size to 64, iterate for 100 epochs, and continue training until the loss converges; ③ During training, the model performance is evaluated on the validation set every 10 epochs, and the model weights with the highest mAP values ​​are saved.

[0037] In summary, the technical solution of this invention consists of two core modules: a cross-modal feature learning module and an auxiliary hash learning module. The overall process is as follows: multimodal data preprocessing → dynamic mask feature filtering → channel-weighted feature optimization → contrastive loss hash code generation → cross-modal hash matching retrieval. These two modules work together to achieve end-to-end learning of "accurate feature extraction—dynamic weight adjustment—optimized hash code generation," effectively solving technical problems such as semantic gap and channel imbalance in existing technologies.

Claims

1. A semantic alignment oriented cross-modal weighted hashing retrieval method, characterized in that, Includes the following steps: Step 1: Achieve accurate extraction and semantic alignment of image and text features through a dynamic masking mechanism; Step 2: Generate high-quality binary hash codes through channel adaptive weighting and contrastive loss optimization; Step 3: Perform cross-modal hash matching retrieval.

2. The method of claim 1, wherein, Step 1 includes the following steps: Step 1-1) Learn image features; specifically including: Sub-step 1-1-1) Perform image segmentation and linear mapping; Sub-step 1-1-2) involves adding location codes; Sub-step 1-1-3) performs feature filtering for the dynamic masking mechanism; Steps 1-2) involve learning text features; specifically including: Sub-step 1-2-1) Encodes text sub-words; Sub-step 1-2-2) performs position encoding and dynamic masking processing; Steps 1-3) The image features obtained in step 1-1) and the text features obtained in step 1-2) are mapped to a unified semantic feature space after being processed by the same encoder and dynamic mask.

3. The method according to claim 2, characterized in that, In sub-step 1-1-1), image data with a size of is input, divided into non-overlapping image blocks with a size of , and the total number of image blocks is calculated; wherein H is the image height, W is the image width, C is the number of image channels, and P is the image block size. Flatten each image patch into a one-dimensional vector to construct the image patch feature matrix ; By a linear transformation matrix The image block features are mapped to a semantic feature space of dimension , by a linear transformation matrix ； in, For the bias term, the mapped features It preserves both the low-level visual features and high-level semantic features of the image.

4. The method according to claim 3, characterized in that, In sub-step 1-1-2), to preserve the spatial location information of the image patch, a learnable location code is added to the mapped features. This yields image features with location information: ； The introduction of location encoding enables the model to distinguish image patches at different locations, improving the spatial resolution of features.

5. The method according to claim 4, characterized in that, In sub-step 1-1-3), to address the issue of overly even weight distribution in traditional attention mechanisms, a dynamic mask matrix is ​​introduced. To filter image features, the mask matrix is ​​defined as follows: ； When the model normally computes the attention weight for that position; When the attention score of the location is completely shielded, and the model only focuses on the key image block features; The position-encoded features are input into the Transformer encoder, and multi-head self-attention is calculated using a dynamic mask matrix, as shown in the formula: ； in, , , These are query, key, and value characteristics, respectively. For key feature dimension, M is the corresponding transformation matrix; M is the mask matrix. This is the transpose of the Key vector; Finally, the image semantic features are obtained through the feedforward neural network MLP processing wherein, is the i-th input feature vector, is the output of the i-th input feature vector, is the output after MLP transformation.

6. The method according to claim 2, characterized in that, In sub-step 1-2-1), the specific steps are as follows: Input text data and perform sub-word segmentation using byte pair encoding (BPE); by statistically analyzing the frequency of character pairs in the text, high-frequency character pairs are merged into new sub-word units, effectively solving the problem of out-of-vocabulary words for rare vocabulary; after segmentation, a sub-word unit of length is generated. The subword sequence is mapped to a dimension of through the embedding layer. Text feature matrix .

7. The method according to any one of claims 2 to 6, characterized in that, In sub-step 1-2-2), the specific steps are as follows: add positional encoding to the text features. ,get To preserve the word order information of the text, the positionally encoded text features are input into the Transformer encoder, which uses the same dynamic masking mechanism as the image features to filter out key semantic features in the text, including core nouns and verbs, and finally outputs the text semantic features. ,in, For the multilayer perceptron to act on the input .

8. The method according to claim 7, characterized in that, Step 2 includes the following steps: Step 2-1) Perform linear dimensionality reduction and nonlinear activation of the features; Step 2-2) Perform adaptive weighting of the channels; Steps 2-3) generate the hash code; In step 2-1), the specific procedure is as follows: Image features and text features The input is a linear dimensionality reduction layer, mapped to a low-dimensional embedding space, reducing computational complexity. The formula is as follows: ； in, For a dimension reduction matrix, Let D×d be a real matrix. For bias terms, It is a d-dimensional real vector. The feature dimension after dimensionality reduction; The ReLU activation function is used to perform a nonlinear transformation on the dimensionality-reduced features, thereby enhancing the expressive power of the features: ； in, Nonlinear and sparsity processing of image features This involves non-linear and sparsity processing of text features.

9. The method according to claim 8, characterized in that, Step 2-2) includes the following steps: Sub-step 2-2-1) Calculate global channel statistics: Features after activation and Perform global average pooling to obtain statistical information for each channel. : ； in, Represents either image (I) or text (T) modality. The spatial dimension of the feature For the number of channels, It is a real vector of dimension C; Sub-step 2-2-2) Calculate the channel weights: Statistical information Two fully connected layers are input to achieve channel-level compression and restoration, capturing the non-linear correlation between channels; the first fully connected layer... Change the number of channels from Compress to , For compression ratio, channel-weighted intermediate features Activated via ReLU: ； in, This is the weight matrix of the first fully connected layer. This is a bias term used to adjust the activation threshold of neurons; This indicates that a nonlinear activation operation is applied to the result of the linear transformation; Second fully connected layer Restore the number of channels to The channel weight vector is obtained. : ； in, , ; The weights are normalized using the Sigmoid function. Interval: ； Sub-step 2-2-3) generates channel-weighted features: Weight vector With features , Channel-by-channel dot product enhances key channel features and suppresses redundant channel features: ； in, This indicates element-wise multiplication. and The final embedded features are the weighted values. In steps 2-3), the specific procedure is as follows: the weighted features are input into the linear transformation layer and mapped to the hash probability space. ； in, Let be the transformation matrix. For bias terms, Representing an image or text modality; calculating the probability of each hash bit being 0 or 1 using the softmax function: ； Generate binary hash codes using the maximum probability method: ； Among them B i This represents the binary hash code of the i-th bit, and the final generated hash code is of length . Hash vector: ； in, This refers to the binary hash code of the k-th candidate sample.

10. The method according to claim 1, 2, 3, 4, 5, 6, 8, or 9, characterized in that, Step 2 also includes steps 2-4) to optimize the contrastive loss function: To improve the discriminative power of hash codes, maximize the similarity between positive samples and images and text, and minimize the similarity between negative samples, a contrastive loss function is designed to constrain the hash code distance between samples of the same class and samples of different classes. 2-4-1) Feature normalization: First, the mapped image feature f i and the text feature f t are denoted as X1 and X2, respectively; L2 normalization is performed on the image and text embedding features to avoid the similarity calculation bias caused by the length difference of the feature vectors: ； in, This represents the global feature vector of the image after encoder and nonlinear transformation, used for subsequent hash mapping or cross-modal matching; This represents the global feature vector of the text after encoder and nonlinear transformation, used for cross-modal matching with image features; 2-4-2) Similarity calculation: Calculate the cosine similarity matrix between normalized features to measure the degree of matching between samples across modalities: ； , This represents the feature representation of the i-th image patch after nonlinear transformation or attention weighting, which is used for subsequent interaction with text features. This represents the feature representation of the j-th text word after nonlinear transformation or attention weighting, used for interaction with image features; 2-4-3) Loss Function Construction: The positive sample loss is defined as minimizing the similarity distance between samples of the same type, and the formula is: ； in, For the label matrix, Indicates the first The image and the first Each text is a matching pair; The negative sample loss is defined as maximizing the similarity distance between out-of-class samples, and the formula is: ； in, This is a similarity threshold used to control the minimum distance between dissimilar samples; The total loss is a weighted sum of the positive and negative sample losses, which serves as the optimization objective of the model. ； By minimizing the total loss, the hash codes generated by the model can achieve the distribution characteristics of samples of the same type being close in distance and samples of different types being far in distance, which significantly improves the accuracy of cross-modal retrieval.

Images