Multi-modal data identifier generation method and system based on semantic hash

Abstract

The invention discloses a multi-modal information identifier generation method and system based on semantic hash. The method comprises the following steps: carrying out data preprocessing and multi-modal feature extraction on multi-modal data to obtain a unified representation containing rich semantic information; mapping the features to a shared semantic space by adopting an independent alignment projection network of each mode, and introducing cross-mode contrast learning and label supervision to realize semantic alignment among different modes; compressing the high-dimensional features after multi-modal data alignment to generate a Hash code with a fixed length; generating a unified semantic hash code containing semantic information of at least two modals; searching the number of times that the Hash code appears in a database through the unified semantic Hash code, generating a redundant code, and splicing the redundant code with the unified semantic Hash code to form an identifier of the sample; on the basis of the unified semantic hash codes, the hash codes related to the to-be-queried category in the test set are queried in the training set, and unified identification and efficient retrieval of different modes are achieved.

Description

Technical Field

[0001] The present invention relates to a multimodal data identification generation method based on semantic hashing, and also relates to a corresponding multimodal data identification generation system, belonging to the field of data processing technology. Background Art

[0002] In today's society, with the rapid development of science and technology, scientific resources (such as scientific literature) and large-scale multimodal data such as multimedia are experiencing explosive growth. Against this backdrop, how to efficiently and accurately retrieve required information from this massive amount of data using cross-modal retrieval technology has become a key issue that is of great concern and urgently needs to be addressed in academia and industry. The core foundation for achieving cross-modal retrieval lies in the unified modeling and fusion encoding of data from different modalities, such as images and text.

[0003] Cross-modal retrieval technology has significant advantages. It can deeply explore the semantic relevance between data of different modalities, thereby effectively improving the semantic expression ability of identification generation and greatly improving computing efficiency. For this reason, this technology has broad application prospects in building intelligent, efficient, and reliable information management systems. Through comprehensive joint processing of multimodal data and scientific hash coding, it can effectively promote the intelligent identification, rapid retrieval, and deep integration and utilization of scientific and technological resources in various modal expressions such as graphics and text, providing strong support for the efficient management and utilization of data in various fields.

[0004] Among the many related research results, the Chinese patent application with application number 202311857379.3 discloses an innovative cross-modal hash retrieval method. This method first obtains the common feature representation and unique feature representation of multimodal data, and constructs a potential data consistency learning model based on these two types of features, striving to ensure the consistent association between different modal data from the data level. At the same time, based on the optimal feature representation of multimodal data, a latent semantic consistency learning model is constructed in the form of an asymmetric paradigm, focusing on deep mining and alignment at the semantic level. During the model training process, the latent data consistency learning model and the latent semantic consistency learning model are iteratively optimized respectively, and the optimized model is used to generate the optimal hash code. Finally, the hash function is learned based on the optimal hash code, and the cross-modal hash retrieval is completed with the help of the learned hash function, realizing fast and accurate matching and retrieval between different modal data, providing a technical solution with innovative and application potential for cross-modal data processing and utilization. Summary of the Invention

[0005] The primary technical problem to be solved by the present invention is to provide a method for generating multimodal data identifiers based on semantic hashing.

[0006] Another technical problem to be solved by the present invention is to provide a multimodal data identification generation system based on semantic hashing.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] According to a first aspect of an embodiment of the present invention, a method for generating a multimodal data identifier based on semantic hashing is provided, comprising the following steps:

[0009] Step S1: Perform data preprocessing and multimodal feature extraction on multimodal data to obtain a unified representation containing rich semantic information;

[0010] Step S2: Use an independent alignment projection network for each modality to map features into a shared semantic space, and introduce cross-modal contrastive learning and label supervision to achieve semantic alignment between different modalities;

[0011] Step S3: compress the high-dimensional features after multimodal data alignment to generate a fixed-length hash code;

[0012] Step S4: Using the Pytorch tool and selecting a preset optimization algorithm, a unified semantic hash code containing semantic information of at least two modalities is generated;

[0013] Step S5: Use the unified semantic hash code to search for the number of occurrences of the hash code in the database, generate a redundant code, and combine it with the unified semantic hash code to form the identifier of the sample;

[0014] Step S6: Based on the unified semantic hash code, the hash code related to the category to be queried in the test set is searched in the training set to achieve multimodal retrieval.

[0015] Preferably, in step S3, the aligned fusion features are input into a hash network, and a semantic hash code is generated through a hash metric loss and a quantization loss function.

[0016] Preferably, step S2 includes the following sub-steps:

[0017] Step S21: The multimodal features obtained in step S1 are respectively input into the alignment projection network of each modality for embedding representation, and converted into feature vectors that can be compared for similarity in the same coordinate system;

[0018] Step S22: using contrastive learning loss function to achieve cross-modal alignment;

[0019] Step S23: Calculate the hash metric loss function To maintain structural consistency within the mode;

[0020] Step S24: Calculate the total alignment loss function.

[0021] Preferably, in step S24, the formula of the total alignment loss function is as follows:

[0022]

[0023] Among them, M i , M j Represents different modes, Represents the mode M i and mode M j Find the contrast loss between them; L im (M k ) represents the mode M k Find the hash metric loss function, λ k In order to adjust the hyperparameter to balance the two losses, the value is set to 0.1; K is the number of modes of multimodal data, and i, j, and k are positive integers less than or equal to K.

[0024] Preferably, the hash metric loss function Satisfy the following formula:

[0025]

[0026] Among them, S ij =1 means samples i and j belong to the same semantic category Otherwise it is 0, which is the supervisory signal of intra-modality consistency;

[0027] d ij =|z i -z j |2 represents the Euclidean distance between samples within or between modalities;

[0028] m1 and m2 are the target distance boundaries of the positive and negative sample pairs, and m1 is smaller than m2.

[0029] Preferably, in step S22, the contrastive learning loss function satisfy:

[0030]

[0031] in, is the cosine similarity function, τ is the temperature coefficient, and B is the size of the subset; and They are respectively aligned representations of images and texts in a shared semantic space.

[0032] Preferably, in step S3, the aligned fusion features are input into the hash network, and the loss is measured by hashing. and quantization loss function Generate semantic hash code:

[0033]

[0034] Among them, p i =(H i +1) / 2 and H i Represents hash network F θ Output the hash vector of sample i, sign is the sign function,

[0035] The overall objective optimization function is Where α is the hyperparameter of the balance coefficient and its value is 10.

[0036] Preferably, the unified semantic hash code in step S4 is expressed as follows:

[0037]

[0038] in, represents the joint modal feature space, F θ Represents a hash network, and sign is a sign function.

[0039] Preferably, in step S5, the value of the redundant code is the number of times the semantic hash code appears.

[0040] According to a second aspect of an embodiment of the present invention, a multimodal data identifier generation system based on semantic hashing is provided, comprising a processor and a memory, wherein the processor and the memory are coupled; wherein the memory is used to store a computer program; and the processor is used to run the computer program stored in the memory to execute the multimodal data identifier generation method based on semantic hashing as described above.

[0041] Compared with existing technologies, this invention achieves deep perception and alignment of semantic information across heterogeneous resources such as text and images by constructing a unified multimodal semantic representation framework, further generating unique, semantic, and efficient multimodal data identifiers. This method not only significantly improves the automation and semantic expression capabilities of identifier generation, but also possesses excellent scalability and cross-domain adaptability, ultimately enabling unified identification, efficient retrieval, and intelligent circulation of multimodal data such as scientific and technological resources across different modalities and institutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a method for generating a multimodal data identifier according to a first embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the alignment projection network in the first embodiment of the present invention;

[0044] Figure 3This is a schematic diagram of the structure of a hash network in the first embodiment of the present invention;

[0045] Figure 4 This is the experimental schematic diagram of the PR curve in the comparative experiment;

[0046] Figure 5 A schematic diagram of the structure of a multimodal data identifier generation system provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical content of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] First embodiment

[0049] like Figures 1 to 3 As shown, the first embodiment of the present invention provides a method for generating a multimodal data identifier based on semantic hashing, which includes at least the following steps:

[0050] Step S1: Perform data preprocessing and multimodal feature extraction on multimodal data to obtain a unified representation containing rich semantic information;

[0051] Step S2: Use an independent alignment projection network for each modality to map features into a shared semantic space, and introduce cross-modal contrastive learning and label supervision to achieve semantic alignment between different modalities;

[0052] Step S3: compress the high-dimensional features after multimodal data alignment to generate a fixed-length hash code;

[0053] Step S4: Use the Pytorch tool and select Adam as the optimization algorithm to generate a unified semantic hash code containing semantic information of at least two modalities;

[0054] Step S5: Use the unified semantic hash code to search for the number of occurrences of the hash code in the database, generate a redundant code, and combine it with the unified semantic hash code to form the identifier of the sample;

[0055] Step S6: Based on the unified semantic hash code, the hash code related to the category to be queried in the test set is searched in the training set to achieve multimodal retrieval.

[0056] To facilitate subsequent explanations, the MS-COCO dataset is used here to construct an image dataset, a text dataset, and a label dataset corresponding to the image-text pairs, and the dataset is divided into a training set and a test set.

[0057] The MSCOCO dataset consists of two subsets: a training set consisting of 82,783 images and a validation set consisting of 40,504 images. In the experiment, the training and validation sets were merged and data samples with missing text descriptions were removed, resulting in 122,218 pairs of images and text (image-text pairs) as experimental data. To partition the training set, 5,000 image-text pairs were randomly selected from the experimental data as the query set, and the remaining samples were used to construct the database. 10,000 samples were randomly sampled from the database for model training, serving as the training set.

[0058] like Figure 1 As shown, the present invention can be used to generate multimodal data identifiers for multimodal data such as text, image, video, and audio. For ease of description, only image and text pairs are used as examples for illustration, but text and video pairs, text and audio pairs, or image and audio pairs can also be used, as long as the data is multimodal data of different modalities (at least two modalities).

[0059] Assume that the training set has N samples (N pairs of images and texts), and the i-th sample is represented as a triplet in represents the image modality, represents the text modality, yi∈{0, 1}C represents the multi-label one-hot encoding of the i-th sample, and C is the total number of labels.

[0060] Step S1: Perform data preprocessing and multimodal feature extraction on multimodal data to obtain a unified representation containing rich semantic information.

[0061] The feature extraction layer provided by the present invention includes an image feature extraction network, a text feature extraction network, a video feature extraction network, and an audio feature extraction network. This step mainly involves two parts: data preprocessing and multimodal feature extraction. For multimodal data (images, text, video, audio), targeted strategies are used to extract high-quality semantic vectors.

[0062] Among them, data preprocessing includes the following sub-steps.

[0063] (1) Outlier detection: Use the Z-score standardization detection method for each numerical field to calculate the standard score of each sample. If an outlier is found, the data record is removed from the training sample.

[0064] (2) Missing value filling: Use the K-nearest neighbor filling algorithm. For each sample with a missing field, find the five closest samples in Euclidean space and average the field value as the filling value;

[0065] (3) Format and structure normalization: For image data, all images are uniformly resized to 224×224, the RGB channel order is maintained, and normalization is performed. For text data, the text information of scientific and technological resources is cleaned. Punctuation marks are removed, characters are converted to lowercase, full-width characters are converted to half-width characters, and stop words are removed. For video data, one key frame is extracted per second (FPS=1), and the longest frame is the first 30 seconds. If the number of key frames exceeds a certain number N=32, frames are extracted at equal intervals; if the number is insufficient, repeated frames are added to fill the gap. For audio data, the audio sampling rate is uniformly 16kHz, with a 25ms frame length and a 10ms frame shift. A 128-dimensional Mel-spectrogram is extracted. If the time window is insufficient, zero padding is performed. If the time window is too long, it is truncated to a fixed length (for example, 5 seconds).

[0066] Among them, multimodal feature extraction includes the following sub-steps.

[0067] (1) Image feature extraction: ResNet-50 is used as the visual feature extractor, and the input is a standardized 224×224 RGB image and the global average pooling layer output after the last convolution layer is intercepted, that is, a 2048-dimensional vector. If a scientific resource has multiple images, 2048-dimensional features are extracted from each image separately, and finally average pooling is performed to output a 2048-dimensional image feature representation. In this embodiment, the VGG convolutional neural network is used to extract the image modality. Extract features and obtain image feature representation where d I Represents the image feature dimension.

[0068] (2) Text feature extraction: Use the BERT-Base model; perform word segmentation on the preprocessed text, and input the segmented text into the model separately. After each text vector is input into the model, the 768-dimensional vector corresponding to the [CLS] position in the output is taken. If the text exceeds the maximum length, the text is truncated at a length of 128. The features extracted after all segmentation are averaged and pooled as the final semantic representation of the entire text modality, with a dimension of 768. In this embodiment, the BERT language model is used to extract the text modality. Extract features and obtain text feature representation where d T Represents text feature dimension.

[0069] (3) Video feature extraction: For each key frame of the video, the semantic feature vector corresponding to each frame is extracted using the image feature extraction method. For the entire key frame sequence, a 2048-dimensional video semantic vector is obtained through time dimension average pooling.

[0070] (4) Audio Feature Extraction: The PANNs-CNN 14 model is used to semantically encode the preprocessed 128-dimensional Mel-spectrogram audio data. When the audio is longer than 5 seconds, a sliding window is used to segment it. 2048-dimensional features are extracted from each segment and then average pooled to obtain an audio representation vector of uniform length.

[0071] Step S2: Use an independent alignment projection network for each modality to map features to a shared semantic space, and introduce cross-modal contrastive learning and label supervision to achieve semantic alignment between different modalities.

[0072] To address the heterogeneity of semantic representation between image and text modalities, this paper designs a joint contrastive learning mechanism, which includes cross-modal contrastive learning loss and intra-modal hash metric loss. It aims to unify the representation of the same semantics in different modalities, improve feature alignment, and lay the foundation for the subsequent generation of unified hash codes.

[0073] The core goal of the alignment projection network, as the semantic alignment layer, is to map the representations of scientific resources across different modalities (images, text, video, and audio) into a shared semantic space. This ensures that the semantic vectors of the same scientific object across all modalities are highly aligned and consistent, thereby supporting cross-modal and intra-modal semantic understanding and retrieval tasks. This layer includes the following three core modules.

[0074] (1) Modality-specific alignment projection network: Each modality (image, text, video, audio) is equipped with an independent alignment projection network to transform the original modality features into a representation with unified semantic dimension and structure.

[0075] Each alignment projection network adopts a three-layer multi-layer perceptron (MLP) structure. The network structure is as follows: Figure 2 As shown in the figure, it specifically includes the following components: Each layer consists of a fully connected layer (Linear), a batch normalization layer (BatchNorm), and an activation function layer (ReLU). The raw features of the modality are used as input, and the features are transformed through the alignment projection network corresponding to the modality. The output vector of the final layer is the representation of the modality in the shared semantic space. The output dimension is maintained at 768 for all modalities.

[0076] (2) InfoNCE loss for cross-modal alignment training: During the training phase, in order to achieve semantic alignment between different modalities, the training objective is constructed using contrastive learning.

[0077] The system collects multiple representations of the same scientific object in different modalities, such as images and text, video and audio, etc. During training, the InfoNCE (Information Noise Contrastive Estimation) loss function is used to bring different modal representations of the same object closer together, while the modal representations of other unrelated objects are pushed further apart. Taking image-text as an example, the goal of the model learning is to make the image and text features describing the same object as close as possible in the shared semantic space, while the image-text pairs describing different objects maintain distance. This cross-modal comparative training method enables the system to gradually establish a semantic alignment relationship between different modalities.

[0078] (3) Label-guided contrastive loss under intra-modality alignment training: In addition to alignment between different modalities, it is also necessary to strengthen semantic aggregation within the same modality. To this end, the semantic alignment layer also introduces intra-modality supervised contrastive loss.

[0079] The specific method is: using the label of each scientific and technological object as a supervisory signal, multiple samples of the same category (or with high label overlap) are brought closer in the shared semantic space, while samples of different categories maintain a certain degree of distinction. For example, in the image modality, if two images are both labeled as "satellite remote sensing" or "photoelectric detection", their extracted semantic vectors will be guided closer. This method improves the clarity of the intra-modal structure in the semantic space through a training mechanism of intra-class clustering and inter-class separation, which is beneficial to subsequent classification, retrieval and other tasks.

[0080] Specifically, this step includes the following sub-steps S21 to S24.

[0081] Step S21: The multimodal features obtained in step S1 (image features and text features in this embodiment) are respectively input into the alignment projection network of each modality for embedding representation, and converted into feature vectors that can be compared for similarity in the same coordinate system.

[0082] Each alignment projection network consists of two layers of multi-layer perceptrons (MLPs) and an output fully connected layer;

[0083] Each layer of MLP consists of a sequentially stacked fully connected layer (FC), batch normalization layer (BN) and ReLU activation function.

[0084] The aligned projection network structure is as follows:

[0085]

[0086] in, and They are respectively aligned representations of images and texts in a shared semantic space. Represents the image feature vector of the i-th image sample, the image features extracted from step S1. The text feature vector representing the i-th text sample is the text feature extracted from step S1. I represents the projection function of the image; P T A projection function representing text. They represent the outputs of each layer of the text fully connected layer; BN represents the batch normalization layer; and ReLU represents the activation function.

[0087] The final feature vector is obtained by transforming the original feature through the projection function according to the above formula In this way, multimodal data from different modalities (images or texts) are mapped into the same shared semantic space using a projection function. In this space, different modal contents with similar semantics (such as "a picture" and "a paragraph of text" describing its content) will be close to each other in the vector space ( dot product or cosine similarity is high). This is conducive to achieving cross-modal tasks such as image-text retrieval and visual question answering.

[0088] Step S22: Use the contrastive learning loss function InfoNCE to achieve cross-modal alignment.

[0089] Contrastive Learning Loss Function satisfy:

[0090]

[0091] in, is the cosine similarity function, τ is the temperature coefficient, and B is the size of the subset (mini-batch).

[0092] The contrastive learning loss function calculates the probability of correctly matching text i given all B candidate texts in an image, and takes the negative logarithm of this probability. The goal is to minimize this loss, that is, to maximize the matching probability. This keeps paired images and texts close together in the common semantic space (high similarity) and keeps unpaired images and texts apart (low similarity).

[0093] Step S23: Calculate the hash metric loss function To maintain structural consistency within the mode.

[0094] Hashing metric loss function Satisfy the following formula:

[0095]

[0096] Among them, S ij =1 means samples i and j belong to the same semantic category Otherwise, it is 0, which is the supervisory signal of intra-modality consistency; d ij =|zi -z j |2 represents the Euclidean distance between samples within or between modalities; m1 and m2 are the target distance boundaries of positive and negative sample pairs, that is, they represent the target distance of samples of the same type (with a smaller expected value) and the target distance of samples of different types (with a larger expected value), respectively. m1 is smaller than m2.

[0097] The hash metric loss function constrains distance relationships in feature space, forcing similar samples to be close together and heterogeneous samples to be separated, thereby maintaining structural consistency within the modality. Each alignment projection network corresponds to a hash metric loss function, and the hash network output of each modality in the hash metric loss function is replaced with the output of the alignment projection network for each modality.

[0098] Step S24: Calculate the total alignment loss function

[0099]

[0100] Among them, M i , M j Represents different modes, Represents the mode M i (Modal M i The aligned projection network output) and the modality M j (Modal M j Compute the contrast loss between the aligned projection network outputs of L im (M k ) represents the mode M k Find the hash metric loss function (modality M k The output of the aligned projection network), λ k To adjust the hyperparameter to balance the two losses, the value is set to 0.1. K is the number of modes of multimodal data, and i, j, and k are positive integers less than or equal to K.

[0101] Since in this step, the original semantic relationship (same / different) of the samples in the modality is obtained by S ij Encoded as a supervisory signal; the hash metric loss function Lm requires the projected feature z i Satisfies: the distance between similar samples ≈ m1 < the distance between heterogeneous samples ≈ m2, so that the distance relationship in the feature space is consistent with the original semantic structure.

[0102] L InfoNCE It is used to align positive sample pairs of different modalities (such as image and text pairing), but may ignore the original relationship between samples within the modality; Lm is introduced as a regularization term to constrain the relative positions of samples within the same modality, avoiding the model from excessively distorting the intra-modal structure (such as compressing all images into one point) in pursuit of cross-modal alignment, that is, avoiding cross-modal alignment from destroying the intra-modal structure.

[0103] m1 and m2 provide flexible boundaries (rather than hard thresholds) to avoid forcing the distance between similar samples to 0 (allowing diversity within the class) and to avoid forcing the distance between heterogeneous samples to be infinite (preventing training instability). ij ) 2 Continuously penalize out-of-bounds behavior, thus preventing model collapse through boundary control.

[0104] It means taking the average of the losses of all unordered sample pairs (i, j) to ensure stability under different batch sizes, that is, using the normalization coefficient to balance the contribution of sample pairs.

[0105] Therefore, the total loss function is used to balance the two losses and achieve the unification of cross-modal alignment and intra-modal consistency.

[0106] In order to improve the generalization ability and semantic preservation ability of multimodal hashing, the present invention introduces cross-modal comparison constraints and intra-modal consistency constraints. Therefore, cross-modal comparison constraints can be used to compress the intrinsic dimension of the data, thereby tightening the generalization error of the hash function, and then by combining cross-modal comparison constraints and intra-modal consistency constraints, the retrieval performance can be improved.

[0107] To demonstrate the technical effectiveness of this step, the following briefly explains the relationship between contrastive loss and hashing performance, drawing on functional complexity theory: Under the premise of intra-modal stability, reducing contrastive loss can effectively improve generalization performance. In other words, combining cross-modal contrast constraints with intra-modal consistency constraints improves retrieval performance.

[0108] Let the class of multimodal hash functions be in Represents the joint modal feature space, which is composed of image features and text features in this embodiment, that is, Represents the target hash code space, indicating that each sample is mapped to a k-dimensional real vector, which can be subsequently converted into a binary code through a sign function.

[0109] According to Dudley's integral bound formula, we have:

[0110]

[0111] in is the empirical Readmacher complexity, and n is the size of the entire data set, that is, the more data, the smaller the generalization error. is the covering number under the norm scale ξ.

[0112] General hash network F θ Meet L F -Lipschitz condition, then:

[0113]

[0114] Assuming that the contrast loss InfoNCE is ε, when the alignment projection network does not change the amount of information within each modality, we have Where A is a constant related to the original distribution of the fused features, is the entropy of the joint feature. The coverage number of the joint feature and its intrinsic dimension r have the following relationship: V(ξ, Z) = Cξ -r , which is related to entropy as follows:

[0115] From the above correlation, we can conclude that the generalization error bound of multimodal hashing performance and the cross-modal contrast loss ε have the following relationship:

[0116]

[0117] Pick Can get

[0118]

[0119] After points are accumulated: According to the relationship between generalization error and Readmacher complexity: in represents the generalization error of the multimodal hash function on the real data set, and δ is the confidence level of the formula.

[0120]

[0121] The above formula shows that when the dataset size is constant, if the aligned projection network does not change the amount of information between the modalities, reducing the contrast loss can effectively improve the generalization performance, which also proves the effectiveness of the contrastive learning loss function.

[0122] As mentioned above, the contrast loss improves generalization performance by relying on the stability within the modality. Therefore, the hash metric loss between modalities can be used to ensure that the basic information used for hashing remains unchanged.

[0123] Here, the image modality feature is set to I i , text mode is T i , whose corresponding label is Y i ∈{0, 1} C , through the modal internal condition constraint (i.e., modal consistency constraint), so that samCategory is a symbol used to indicate whether two samples belong to the same category, that is, to maintain the consistency of the internal geometric relationship of the modality in the feature space and make it consistent with the label.

[0124] Step S3: Compress the high-dimensional features after multimodal data alignment to generate a fixed-length semantic hash code (including a 64-bit semantic main code and a 16-bit redundant code).

[0125] The aligned fusion features are input into the hash network and the loss is measured by hashing. and quantization loss function Generate semantic hash code:

[0126]

[0127] Among them, p i =(H i +1) / 2 and H i Represents hash network F θ Output the hash vector of sample i, sign is the sign function.

[0128] The overall objective optimization function is Where α is the hyperparameter of the balance coefficient and its value is 10.

[0129] Hash vector for input (example):

[0130]

[0131] The following semantic hash code is obtained (example).

[0132]

[0133] Step S4: Use the Pytorch tool and select Adam as the optimization algorithm to generate a unified semantic hash code containing semantic information of at least two modalities (image and text):

[0134]

[0135] Therefore, the images of all samples in the test set are sent to the VGG convolutional neural network to obtain image features; the texts of all samples are passed through the BERT model to obtain text features; the image features and text features are used to obtain a unified semantic hash code containing images and texts in the test set according to the method of steps S1 and S4 of the training process.

[0136] Although the above description uses images and text as examples, and uses steps S1 to S4 to obtain unified semantic hash codes for images and text, for samples containing images, text, audio, and video, the above steps S1 to S4 can also be used to obtain unified semantic hash codes for images, text, audio, and video, which provides a retrieval basis for cross-modal retrieval.

[0137] Step S5: Use the unified semantic hash code to search for the number of occurrences of the hash code in the database, generate a redundant code, and combine it with the unified semantic hash code to form the identifier of the sample.

[0138] In this embodiment, the redundant code is a 16-bit binary number, and its value is the number of times the semantic hash code appears.

[0139] Step S6: Based on the unified semantic hash code, the identifiers related to the category to be queried in the test set are searched in the training set to achieve multimodal retrieval.

[0140] To verify the technical effects of the embodiments of the present invention, training and validation were performed on the MSCOCO dataset, which consists of a training set of 82,783 images and a validation set of 40,504 images. In the experiment, the training and validation sets were merged, and data samples with missing text descriptions were removed, resulting in 122,218 pairs of image and text samples as experimental data. For sample partitioning, 5,000 pairs of image and text samples were randomly selected as the query set, and the remaining samples were used to construct the database. A further 10,000 samples were randomly sampled from the database for model training.

[0141] This paper uses a VGG convolutional neural network to extract 4096-dimensional image features and a BERT convolutional neural network to extract 2000-dimensional text features. The mean average precision (mAP@All) is used as the performance evaluation metric, where All represents the mAP value calculated from the entire database query sort results. This scheme is compared with the GCIMH method (Xiaobo Shen, Yinfan Chen, Shirui Pan, Weiwei Liu, and Yuhui Zheng. 2023. Graph convolutional incomplete multi-modal hashing. In Proceedings of the 31st ACM international conference on multimedia. 7029-7037.) and the ACMVL method (Jian Zhu, Lei Liu, Yu Zhang, Chang Tang, and Li-Rong Dai. 2025. Adaptive Confidence Multi-View Learning. IEEE Transactions on Multimedia (2025)). The results of 16-bit, 32-bit, 64-bit, and 128-bit multimodal retrieval tasks are shown in Table 1. The present invention performs better.

[0142] Table 1 mAP performance experiment table

[0143] Method 16 bits 32 bits 64 bits 128 bits ACMVL 0.5938 0.6321 0.6567 0.6752 GCIMH 0.6590 0.7007 0.7205 0.7374 Ours 0.6922 0.7378 0.7844 0.7943

[0144] Figure 4 A clearer effect is provided, in which the present invention is represented by the blue line. When the recall rate is less than 0.9, it outperforms the ACMVL method and the GCIMH method.

[0145] Compared with the existing technology, the present invention proposes an innovative multimodal hash learning framework, which aims to improve the effect of multimodal retrieval. First, an enhancement mechanism based on multimodal feature alignment is designed to effectively fuse multiple modal information such as images and texts, and achieve deep complementarity at the semantic level. Secondly, the joint constraints of cross-modal information entropy and intrinsic manifold structure are introduced, so that the fused features have lower intrinsic dimensions and stronger semantic consistency. Finally, the hash network is trained by satisfying hash metric loss and quantization loss to ensure the smooth mapping and good generalization ability of the hash function. Experimental results show that the present invention is significantly better than existing advanced methods on multiple mainstream data sets, reflecting its advantages in maintaining global semantic structure and improving cross-modal retrieval performance.

[0146] Second embodiment

[0147] like Figure 5 As shown, based on the above-mentioned multimodal data identification generation method based on semantic hashing, the second embodiment of the present invention further provides a multimodal data identification generation system based on semantic hashing. The multimodal data identification generation system includes one or more processors and a memory. The memory is coupled to the processor and is used to store one or more programs. When the program is executed by the processor, the processor implements the multimodal data identification generation method based on semantic hashing as in the above-mentioned embodiment.

[0148] Among them, the processor is used to control the overall operation of the multimodal data identification generation system to complete all or part of the steps of the multimodal data identification generation method based on semantic hashing. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processing (DSP) chip, etc. The memory is used to store various types of data to support the operation of the multimodal data identification generation system. These data may include, for example, instructions for any application or method used to operate on the multimodal data identification generation system, as well as application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, etc.

[0149] In another exemplary embodiment, the present invention further provides a computer-readable storage medium comprising program instructions, which, when executed by a processor, implement the steps of the method for generating a multimodal data identifier based on semantic hashing in any of the above-described embodiments. For example, the computer-readable storage medium may be the aforementioned memory comprising the program instructions, which may be executed by a system processor to perform the above-described method for generating a multimodal data identifier based on semantic hashing and achieve the same technical effects as the above-described method.

[0150] It should be noted that the above-mentioned multiple embodiments are merely examples, and the technical solutions of the various embodiments can be combined, and the order of the steps can be changed, all within the scope of protection of the present invention.

[0151] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0152] The above describes in detail the semantic hash-based multimodal data identifier generation method and system provided by the present invention. For those skilled in the art, any obvious modification made to the present invention without departing from the essence of the present invention will constitute an infringement of the patent rights of the present invention and will result in corresponding legal liability.

Claims

1. A method for generating multimodal information identifiers based on semantic hashing, characterized in that include: Step S1: Perform data preprocessing and multimodal feature extraction on multimodal data to obtain a unified representation containing rich semantic information; Step S2: Use an independent alignment projection network for each modality to map features into a shared semantic space, and introduce cross-modal contrastive learning and label supervision to achieve semantic alignment between different modalities; Step S3: compress the high-dimensional features after multimodal data alignment to generate a fixed-length hash code; Step S4: Using the Pytorch tool and selecting a preset optimization algorithm, a unified semantic hash code containing semantic information of at least two modalities is generated; S5: Use the unified semantic hash code to find the number of occurrences of the hash code in the database, generate a redundant code, and concatenate it with the unified semantic hash code to form the identifier of the sample; S6: Based on the unified semantic hash code, the hash code related to the category to be queried in the test set is searched in the training set to achieve multimodal retrieval.

2. The method for generating a multimodal information identifier based on semantic hashing according to claim 1, wherein: In step S3, the aligned fusion features are input into the hash network, and a semantic hash code is generated through the hash metric loss and quantization loss function.

3. The method for generating a multimodal information identifier based on semantic hashing according to claim 1, characterized in that The step S2 includes the following sub-steps: Step S21: The multimodal features obtained in step S1 are respectively input into the alignment projection network of each modality for embedding representation, and converted into feature vectors that can be compared for similarity in the same coordinate system; Step S22: using contrastive learning loss function to achieve cross-modal alignment; Step S23: Calculate the hash metric loss function To maintain structural consistency within the mode; Step 24: Calculate the total alignment loss function.

4. The method for generating a multimodal information identifier based on semantic hashing according to claim 3, characterized in that In step S24, the total alignment loss function is expressed as follows: Among them, M i , M j Represents different modes, Represents the mode M i and mode M j Find the contrast loss between them; L im (M k ) represents the mode M k Find the hash metric loss function, λ k In order to adjust the hyperparameter to balance the two losses, the value is set to 0.1; K is the number of modes of multimodal data, and i, j, and k are positive integers less than or equal to K.

5. The method for generating a multimodal information identifier based on semantic hashing according to claim 4, characterized in that The hash metric loss function Satisfy the following formula: Among them, S ij =1 means samples i and j belong to the same semantic category Otherwise, it is 0, which is the supervisory signal of intra-modality consistency; d ij =|z i -z j |2 represents the Euclidean distance between samples within or between modalities; m1 and m2 are the target distance boundaries of positive and negative sample pairs, and m1 is smaller than m2.

6. The method for generating a multimodal information identifier based on semantic hashing according to claim 5, characterized in that In step S22, the contrastive learning loss function Satisfy the following formula: in, is the cosine similarity function, τ is the temperature coefficient, and B is the size of the subset; and They are respectively aligned representations of images and texts in a shared semantic space.

7. The method for generating a multimodal information identifier based on semantic hashing according to claim 6, characterized in that In step S3, the aligned fusion features are input into the hash network, and the loss is measured by hashing. and quantization loss function Generate semantic hash code: Among them, p i =(H i +1) / 2 and H i Represents hash network F θ Output the hash vector of sample i, sign is the sign function, and the overall target optimization function is Where α is a hyperparameter of the balance coefficient.

8. The method for generating a multimodal information identifier based on semantic hashing according to claim 7, characterized in that The unified semantic hash code in step S4 is expressed as follows: b i =sing(F θ (Z i )), in, represents the joint modal feature space, F θ Represents a hash network, and sign is a sign function.

9. The method for generating a multimodal information identifier based on semantic hashing according to claim 8, characterized in that In step S5, the value of the redundant code is the number of times the semantic hash code appears.

10. A multimodal information identification generation system based on semantic hashing, characterized by It includes a processor and a memory, the processor and the memory are coupled; wherein the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute the multimodal information identification generation method based on semantic hashing as described in any one of claims 1 to 9.

Images