Multi-mode Hash retrieval method, system and equipment based on multi-view center structure and medium

Abstract

The invention discloses a multi-modal Hash retrieval method, system and equipment based on a multi-view center structure and a medium, belongs to the technical field of artificial intelligence and multi-modal Hash retrieval, and aims at achieving the technical purpose of obtaining balance between intra-class compactness and inter-class separability in the multi-modal Hash retrieval process. According to the technical scheme, the method comprises the following steps: constructing a multi-modal data set; constructing a multi-modal Hash retrieval model based on a multi-view center structure; training the model; wherein the construction of the multi-modal Hash retrieval model based on the multi-view center structure specifically comprises the following steps: modal specific prototype learning: respectively extracting refined features from corresponding modals by using an image modal depth multilayer perceptron and a text modal depth multilayer perceptron; calculating the average value of the refined features of the image modality and the text modality to obtain a modality specific prototype, and further obtaining the unique features of the image modality and the unique features of the text modality; carrying out multi-modal integrated prototype learning; and performing multi-view semantic enhanced hash learning.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and multimodal hash retrieval, and in particular to a multimodal hash retrieval method, system, device and medium based on a multi-view center structure. Background Art

[0002] With the continuous advancement of science and technology and the advent of the information age, multimodal data with multiple features such as images, text, video, and audio is widely used, and related data is showing explosive growth. However, the progress of data retrieval methods is relatively lagging behind, and traditional modal data retrieval methods can no longer meet the increasingly diverse needs of multimodal data retrieval. In recent years, the increasing amount of multimodal data and the urgent need for efficient retrieval have promoted the increasing interest in multimodal retrieval research. Multimodal data has the characteristics of correlation and complementarity, that is, data of different modalities have different characteristics, structures, and representations, but they may describe the same entity or event from the perspective of their respective modalities, and there are one or more associations between the data within the modality. Therefore, multimodal data retrieval needs to be able to find relevant information across different modalities, and to ensure its effectiveness within the modality to achieve accurate and efficient retrieval.

[0003] Unsupervised multimodal hashing is an information retrieval technique that aims to map data from different modalities into a common binary space, converting high-dimensional multimodal data into compact and information-rich binary codes in order to achieve efficient retrieval while maintaining the intrinsic semantic structure of the data, achieving cost-effective and computationally efficient multimodal retrieval. Unsupervised multimodal hashing achieves similarity preservation between different modalities without the need for additional labels or semantic information, thereby reducing cost and effort and providing greater flexibility in practical applications.

[0004] In recent years, many multimodal hash retrieval models have been applied. Among them, an unsupervised cross-modal hash retrieval method that integrates multiple instance relationships proposes a new framework structure. This method constructs a similarity matrix based on the instances in each training batch, and processes these relationship graphs through graph reasoning methods to enhance the semantic associations between instances, showing good performance; the deep graph neighborhood consistency preserving network is another unsupervised cross-modal hashing method, which considers the graph neighbor relationship between data and its neighbors, retains the real-valued information of the data through semi-real and semi-binary optimization strategies and converts it into a binary hash code, which is suitable for cross-modal retrieval on large-scale datasets; the deep unsupervised hashing method based on latent semantic components proposes a novel deep semantic component hashing framework. This method regards semantic components as latent variables under the expectation maximization algorithm framework, and designs an iterative algorithm with the maximum likelihood of training data as the goal, which helps the hash model achieve excellent performance. Although multimodal hashing methods have made significant progress, the multimodal fusion methods in the above methods still have problems such as coarse-grained feature limitations, over-simplified fusion strategies and neglect of structural information. Existing work still faces major challenges in ensuring intra-class compactness and inter-class separability. Models generally exhibit obvious inter-class overlap and intra-class dispersion problems. Summary of the invention

[0005] The technical task of the present invention is to provide a multimodal hash retrieval method, system, device and medium based on a multi-view center structure to solve the problem of achieving a balance between intra-class compactness and inter-class separability in the multimodal hash retrieval process.

[0006] The technical task of the present invention is achieved in the following way: a multimodal hash retrieval method based on a multi-view center structure, the method is specifically as follows:

[0007] Constructing a multimodal dataset: Obtain image modality and text modality datasets, process them, and then perform deep feature extraction on them using the pre-trained model BLIP-2 to obtain deep image features and deep text features. At the same time, construct training sets, test sets, and retrieval sets in proportion, and then combine them to generate the final dataset.

[0008] Constructing a multimodal hash retrieval model based on a multi-view center structure: Constructing a multimodal hash retrieval model based on a multi-view center structure through neural network and deep learning technology; wherein, the multimodal hash retrieval model based on a multi-view center structure refers to a multimodal hash model with a prototype as the center structure;

[0009] Training model: Use the final dataset to train the multimodal hash retrieval model based on the multi-view center structure through a composite loss function;

[0010] Among them, the construction of a multimodal hash retrieval model based on a multi-view center structure is as follows:

[0011] Modality-specific prototype learning: Use image modality deep multilayer perceptron and text modality deep multilayer perceptron to extract refined features from the corresponding modalities respectively, and calculate the average value of the image modality and text modality refined features to obtain the modality-specific prototype, and then obtain the unique features of the image modality and the unique features of the text modality, and then realize prototype alignment and prototype separation through the prototype contrast loss function;

[0012] Multimodal ensemble prototype learning: The heterogeneous modal feature set is converted into a unified relaxed hash coding space, and the refined features of the image modality and the text modality are merged using a connection operation. A linear transformation layer is applied to project and adjust the scale of the features in the feature space. The refined joint representation after adjustment is mapped into a continuous hash code containing rich multimodal semantic information through the Tanh activation function, and then the integrated prototype cross entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance to other prototypes.

[0013] Multi-view semantically enhanced hash learning: define a similarity matrix and use the pairwise semantic similarity information to optimize the generation of hash codes through a similarity-preserving loss function. Introduce a structural similarity matrix and use the structural similarity information to optimize the generation of hash codes through a structured neighborhood-preserving loss function to capture the structural information in multimodal data.

[0014] As a preference, the multimodal dataset is constructed as follows:

[0015] Obtain public datasets: Obtain and download public datasets containing image modalities and text modalities from public network resources as original datasets; public datasets include MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset;

[0016] Divide the downloaded original data set;

[0017] Use the pre-trained model BLIP-2 as the image encoder to extract deep features from the image modality and obtain deep image features;

[0018] Use the pre-trained model BLIP-2 as the text encoder to perform deep feature extraction on the text modality and obtain deep text features;

[0019] The data division of the downloaded original data set is as follows:

[0020] Randomly select any part of image-text pairs on the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset as the test set to evaluate the test performance;

[0021] Excluding the test set image-text pairs in the MirFlickr-25K dataset, MS COCO, and NUS-SIDE dataset, a portion of the remaining image-text pairs is randomly selected as the retrieval dataset for the retrieval operation;

[0022] In the retrieval dataset, the remaining image-text pairs in the dataset are used as training sets for objective function training.

[0023] Preferably, the depth image features are obtained as follows:

[0024] Preprocessing image modality data: uniformly crop and reshape the input image to H×H pixel size, then convert the reshaped image into a flattened two-dimensional plane image block, and normalize the flattened two-dimensional plane image block to eliminate the difference between images;

[0025] Extract deep image features: Load the blip2-vision-model model onto the GPU through the pre-trained model BLIP-2, and use the blip2-vision-model model to extract image deep features, thereby capturing high-level semantic information of the image;

[0026] The details of obtaining deep text features are as follows:

[0027] Preprocessing text modal data: preprocessing the input text by text cleaning, text segmentation, and text vectorization;

[0028] Extract deep text features: Load the blip2-vision-model model onto the GPU through the pre-trained model BLIP-2, and use the blip2-vision-model model to extract text deep features to capture high-level semantic information of the text.

[0029] Preferably, the modality-specific prototype learning is as follows:

[0030] Refine the image modality features: The high-level semantic information of the image is passed through a deep multi-layer perceptron specific to the image modality to obtain the image refinement features and map them into a common 512-dimensional space. The formula is as follows:

[0031]

[0032] in, represents the refined feature matrix extracted from the image modality; x represents the xth sample; X represents the input image modality data; θx represents the xth trainable parameter; represents the real number domain; k×m means that the matrix has k rows and m columns. After being processed by the MLP function, the features will be transformed into a space with the same dimension m;

[0033] Refined text modality features: The high-level semantic information of the text is passed through a deep multi-layer perceptron specific to the text modality to obtain image refinement features and map them into a common 512-dimensional space. The formula is as follows:

[0034]

[0035] in, represents the refined feature matrix extracted from the image modality; y represents the yth sample; Y represents the input image modality data; θ y represents the yth trainable parameter; Represents the real number domain; k×m means that the matrix has k rows and m columns. After being processed by the MLP function, the features will be converted into a space with the same dimension m; Deep Multilayer Perceptron MLP x and MLP y They all contain an input layer, a series of carefully designed hidden layers, and an output layer. In order to enhance the generalization ability of the model and alleviate the overfitting problem, a powerful regularization technique, Dropout, is integrated into the model architecture.

[0036] Obtaining image modality-specific prototypes: The specific prototype generator is used to calculate the average value of the refined features of the text modality samples to obtain the modality-specific prototype. The formula is as follows:

[0037]

[0038] Among them, N c Indicates the number of samples belonging to the cth category; function The value is 1 when the i sample belongs to the cth category, otherwise it is 0;

[0039] Obtaining a text modality-specific prototype: The specific prototype generator is used to calculate the average value of the refined features of the image modality samples to obtain the modality-specific prototype. The formula is as follows:

[0040]

[0041] Among them, N c Indicates the number of samples belonging to the cth category; function The value is 1 when the i sample belongs to the cth category, otherwise it is 0;

[0042] The prototype contrast loss function is introduced to perform prototype learning and realize prototype alignment and prototype separation;

[0043] Among them, the prototype contrast loss function is conceptually decomposed into two key components: prototype alignment and prototype separation, and the formula is as follows:

[0044]

[0045] The prototype alignment part makes the prototypes of the same category closer across modalities: the model is optimized by minimizing the distance between the input prototype and the output feature, so that the optimized model can better learn the feature representation of each category. The dot product between the feature vector of category C and the model prediction vector, after adjustment by the temperature parameter, reflects the model's confidence in the current category;

[0046] The prototype separation part separates prototypes of different categories: the model is optimized by maximizing the distance between different categories, so that the optimized model can better distinguish different categories. It is a logarithmic summation operation that involves summing the exponential scores of all non-C categories and then taking the logarithm, which encourages the optimization model to distinguish the feature vector of the current category from the feature vector of non-current categories.

[0047] As a preferred method, the multimodal integrated prototype learning is as follows:

[0048] Refined joint representation: The refined features of the image modality and the text modality are combined using a concatenation operation, and a linear transformation layer is applied to project and rescale the features in the feature space. The formula is as follows:

[0049]

[0050] Among them, Z represents the fused feature matrix; and represents the refined feature matrix extracted from each modality; ⊕ represents the connection operation; W represents the weight matrix; b represents the bias vector;

[0051] Mapping hash code: Applying the Tanh activation function to the refined joint representation maps the refined joint representation into a continuous hash code containing rich multimodal semantic information. The formula is as follows:

[0052]

[0053] Among them, Z represents the representation after refinement and union, Tanh represents the nonlinear hyperbolic tangent function;

[0054] Define multimodal integrated prototype learning, the formula is as follows:

[0055] P c ={p1,p1,…,p C}∈R k×C ;

[0056] Among them, p i represents the corresponding prototype; k represents the total number of prototypes;

[0057] Introducing the integrated prototype cross entropy loss function: The integrated prototype cross entropy loss function is introduced to optimize the model parameters. The integrated prototype cross entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance with other prototypes.

[0058] As a preferred method, multi-view semantic enhanced hash learning is as follows:

[0059] Define a similarity matrix: Use pairwise semantic similarity information as a guide when generating hash codes and introduce a similarity preservation loss function L s , the similarity matrix is ​​integrated into the learning of hash codes to ensure that the generated hash codes can reflect the semantic similarity between samples. The similarity matrix function formula is as follows:

[0060]

[0061] Among them, S ij Represents the similarity matrix, which is used to quantify the semantic similarity between sample i and sample j, l i and l j is the eigenvector of sample i and sample j; e represents the base of natural logarithm, and T represents transpose;

[0062] Define a structural similarity matrix: Introduce the structured neighborhood preservation loss function L q The structural information in multimodal data is captured, and the structural similarity matrix is ​​introduced into the learning of hash codes. By minimizing the structured neighborhood preservation loss function, the hash codes of similar samples are made close to each other, while avoiding excessive similarity between dissimilar samples. The structural similarity matrix function formula is as follows:

[0063]

[0064] Among them, T ij represents the structural similarity matrix, which is used to quantify the similarity of the consecutive hash codes between sample i and sample j; h i and h j represents the consecutive hash codes of sample i and sample j; ||h i ||2||h j ||2 represents the hash code h i and h j The Euclidean norm of ;

[0065] The structural similarity matrix is ​​calculated as the normalized dot product of consecutive hash codes, and its values ​​are adjusted between 0 and 1 to ensure that the values ​​are in a meaningful range.

[0066] Preferably, the training model is as follows:

[0067] Constructing loss function: The goal of the multimodal hash retrieval model based on the multi-view center structure is to construct a loss function to train the multimodal hash retrieval model based on the multi-view center structure by integrating the prototype contrast loss function, the integrated prototype cross entropy loss function, the similarity preservation loss function and the structured neighborhood preservation loss function into a composite function, as follows:

[0068] Minimize the prototype contrast loss function L c , the formula is as follows:

[0069]

[0070] Among them, C represents the total number of categories of all samples; the logarithmic part is the softmax function, which is used to convert the input linear score into a probability distribution and Represent the input feature and output feature respectively; τ is a temperature parameter that controls the scale of the distribution; the temperature parameter controls the smoothness of the softmax function, thereby affecting the generalization ability of the model. The smaller the value of the temperature parameter, the sharper the distribution of the output of the softmax function, that is, the higher the confidence of the multimodal hash retrieval model based on the multi-view center structure for any category; conversely, the larger the value of the temperature parameter, the smoother the distribution of the output of the softmax function, and the more uncertain the prediction of the multimodal hash retrieval model based on the multi-view center structure;

[0071] Minimize the prototype cross entropy loss function, the formula is as follows:

[0072]

[0073] Where D represents the measured Euclidean distance; N represents the number of samples; p i Represents the corresponding prototype; the logarithm is a softmax operation, which is used to calculate the probability that the i-th sample belongs to its matching prototype. In this way, the multimodal hash retrieval model based on the multi-view center structure learns the relative distance between each sample and its matching prototype;

[0074] Minimize the similarity preserving loss function L s , the formula is as follows:

[0075]

[0076] Among them, ||·|| represents the L2 norm; cos(h i ,h j ) represents the continuous hash codes h of sample i and sample j i and hj The cosine similarity between ij represents the similarity matrix;

[0077] Minimize the structured neighborhood preserving loss function L q , the formula is as follows:

[0078]

[0079] Where M represents the number of samples involved in the retrieval process; [Δ] ij represents a binary indicator, which is equal to 1 if samples i and j belong to the same category and 0 otherwise; the structural similarity matrix is ​​calculated as the normalized dot product of consecutive hash codes, adjusted to ensure that the values ​​are in a meaningful range, i.e., between 0 and 1;

[0080] The prototype contrast loss function, the integrated prototype cross entropy loss function, the similarity preservation loss function and the structured neighborhood preservation loss function are integrated into a composite loss function, which is minimized to obtain high-quality hash codes and achieve more efficient multimodal hash retrieval. The specific function is as follows:

[0081] L=αL c +βL s +λL p +ηL q ;

[0082] Among them, α, β, λ, and η are hyperparameters used to balance the contribution of each loss term to achieve the best balance;

[0083] Optimization model: The Adam algorithm is used as the optimization function of the multimodal hash retrieval model based on the multi-view center structure; the learning rate parameter is set to 0.0001, and other hyperparameters use the default values ​​in PyTorch.

[0084] A multi-modal hash retrieval system based on a multi-view center structure, the system is used to implement the multi-modal hash retrieval method based on a multi-view center structure as described above; the system comprises:

[0085] The multimodal dataset construction module is used to obtain image modality and text modality datasets, process the image modality and text modality datasets, and then perform deep feature extraction on the image modality and text modality datasets through the pre-trained model BLIP-2 to obtain deep image features and deep text features; at the same time, the training set, test set and retrieval set are constructed according to the proportion, and then combined to generate the final dataset;

[0086] A multimodal hash retrieval model construction module based on a multi-view center structure is used to construct a multimodal hash retrieval model based on a multi-view center structure through a neural network and deep learning technology; wherein the multimodal hash retrieval model based on a multi-view center structure refers to a multimodal hash model with a prototype as the center structure;

[0087] A model training module is used to train a multimodal hash retrieval model based on a multi-view center structure using the final data set through a composite loss function;

[0088] Among them, the multimodal hash retrieval model construction module based on the multi-view center structure includes:

[0089] The modality-specific prototype learning submodule is used to extract refined features from the corresponding modalities using the image modality deep multilayer perceptron and the text modality deep multilayer perceptron respectively, and calculate the average value of the image modality and text modality refined features to obtain the modality-specific prototype, and then obtain the unique features of the image modality and the unique features of the text modality, and then realize prototype alignment and prototype separation through the prototype contrast loss function;

[0090] The multimodal integrated prototype learning submodule is used to transform the heterogeneous modal feature set into a unified relaxed hash coding space, merge the refined features of the image modality and the text modality using a connection operation, apply a linear transformation layer to project in the feature space and adjust the scale of the features, and map the adjusted refined joint representation into a continuous hash code containing rich multimodal semantic information through the Tanh activation function. The integrated prototype cross entropy loss function is then used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance to other prototypes.

[0091] The multi-view semantically enhanced hash learning submodule is used to define a similarity matrix and utilize the pairwise semantic similarity information to optimize the generation of hash codes through a similarity-preserving loss function. A structural similarity matrix is ​​introduced and the structural similarity information is utilized to optimize the generation of hash codes through a structured neighborhood-preserving loss function to capture the structural information in multimodal data.

[0092] An electronic device comprising: a memory and at least one processor;

[0093] Wherein, the memory stores a computer program;

[0094] The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the multimodal hash retrieval method based on the multi-view center structure as described above.

[0095] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the multimodal hash retrieval method based on a multi-view center structure as described above.

[0096] The multi-modal hash retrieval method, system, device and medium based on the multi-view center structure of the present invention have the following advantages:

[0097] (i) The present invention solves the problem of the difficulty in achieving a balance between intra-class compactness and inter-class separability in existing work. The prototype of a specific modality is learned through intra-modal class information to ensure the effective fusion of the unique features of each modality, and a cross-modal multimodal integration class prototype is learned, thereby significantly enhancing the inter-class separability and solving the problems of inter-class overlap and intra-class dispersion in multimodal hashing, thereby ensuring a balance between intra-class compactness and inter-class separability.

[0098] (ii) This invention addresses the limitations of traditional methods by identifying the importance of local nuances within a single modality and global information across multiple modalities, and achieves a significant improvement in retrieval performance compared to existing state-of-the-art methods; its core idea is to seamlessly integrate the advantages of modality prototypes and integrated prototypes and comprehensively explore semantic information from multiple perspectives;

[0099] (iii) The present invention proposes a new multimodal hashing framework, which first learns a set of continuous hash codes that are optimized in a continuous space to maintain the semantic similarity between multimodal samples, and then applies a sign function to the set of continuous hash codes to obtain strict binary hash codes, which ensures that the generated binary hash codes not only effectively compress high-dimensional multimodal samples, but also maintain the ability to distinguish to achieve accurate retrieval;

[0100] (iv) The present invention uses modality-specific deep multilayer perceptrons to recognize that each modality in heterogeneous data carries its own unique modality-specific information, extracts refined features from each modality, and more carefully captures the subtle properties and features inherent in different data modalities, ensuring that the unique information contained in each modality is fully utilized. These modality-specific deep multilayer perceptrons are used to map features from different modalities into a common unified feature space. This transformation simplifies the process of heterogeneous modality alignment and paves the way for complex and effective fusion of heterogeneous modalities.

[0101] (V) The present invention proposes a method that utilizes the advantages of modality-specific prototypes and integrated prototypes to seamlessly integrate local and global class information to promote the combination of intra-modality and inter-modality correlations. The method mainly consists of two parts, namely prototype alignment and prototype separation. The former aims to make prototypes of the same category across modalities closer (prototype alignment), while the latter pushes prototypes of different categories apart (prototype separation). By effectively utilizing these prototypes, the semantic depth and discrimination ability of the hash code are significantly enhanced.

[0102] (VI) The present invention proposes a prototype contrast loss function, which effectively aligns prototypes across modalities within a class by optimizing the prototype contrast loss, while separating prototypes of different categories;

[0103] (VII) The present invention proposes an integrated prototype cross entropy loss function, which minimizes the distance between the hash code and its matching prototype, while maximizing the distance to other prototypes, promoting intra-class cohesion while enhancing inter-class separation;

[0104] (VIII) The present invention proposes a similarity preserving loss function, which integrates the similarity matrix into the learning of hash codes and uses paired semantic similarity information to enhance the discrimination ability of hash codes;

[0105] (IX) The present invention proposes a structured neighborhood preservation loss function, which aims to minimize the ambiguity between adjacent samples, ensure that the generated hash code accurately maintains the similarity structure, promotes the hash codes of similar samples to be close to each other, and avoids excessive similarity between dissimilar samples;

[0106] (J) The present invention proposes to optimize the generation process of hash codes by minimizing a composite loss function, thereby solving the limitations brought by coarse-grained feature limitations, simplified fusion strategies and neglect of structural information, and enhancing the retrieval performance of multimodal hashing. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] The present invention is further described below in conjunction with the accompanying drawings.

[0108] Attached Figure 1 It is a flowchart of a multimodal hash retrieval method based on a multi-view center structure;

[0109] Attached Figure 2 A flowchart for building a multimodal hash retrieval model based on a multi-view center structure;

[0110] Attached Figure 3 This is a flowchart for training the model;

[0111] Attached Figure 4 A flowchart for constructing a multimodal dataset;

[0112] Attached Figure 5A flowchart for modality-specific prototype learning;

[0113] Attached Figure 6 A flowchart for prototyping learning for multimodal integration;

[0114] Attached Figure 7 Flowchart of hashing learning for multi-view semantic enhancement. DETAILED DESCRIPTION

[0115] The multimodal hash retrieval method, system, device and medium based on a multi-view center structure of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0116] Embodiment 1:

[0117] As attached Figure 1 As shown, this embodiment provides a multi-modal hash retrieval method based on a multi-view center structure, and the method is specifically as follows:

[0118] S1. Construct a multimodal dataset: obtain image modality and text modality datasets, process them, and then perform deep feature extraction on the image modality and text modality datasets through the pre-trained model BLIP-2 to obtain deep image features and deep text features; at the same time, construct a training set, a test set, and a retrieval set in proportion, and then combine them to generate the final dataset;

[0119] S2. Constructing a multimodal hash retrieval model based on a multi-view center structure: constructing a multimodal hash retrieval model based on a multi-view center structure through neural network and deep learning technology; wherein, the multimodal hash retrieval model based on a multi-view center structure refers to a multimodal hash model with a prototype as the center structure;

[0120] S3. Training model: Use the final dataset to train the multimodal hash retrieval model based on the multi-view center structure through a composite loss function.

[0121] As attached Figure 4 As shown, the construction of the multimodal dataset in step S1 of this embodiment is specifically as follows:

[0122] S101, obtaining public datasets: obtaining and downloading public datasets including image modalities and text modalities from public network resources as original datasets; wherein the public datasets include MirFlickr-25K datasets, MS COCO and NUS-SIDE datasets;

[0123] S102, dividing the downloaded original data set;

[0124] S103, using the pre-trained model BLIP-2 as an image encoder, performing deep feature extraction on the image modality to obtain deep image features;

[0125] S104: Use the pre-trained model BLIP-2 as a text encoder to perform deep feature extraction on the text modality to obtain deep text features.

[0126] In step S102 of this embodiment, the data division of the downloaded original data set is specifically as follows:

[0127] S10201. Randomly select any part of image-text pairs on the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset as the test set to evaluate the test performance.

[0128] S10202, exclude the test set image-text pair data in the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset, and randomly select a part of the remaining image-text pairs as the retrieval dataset for the retrieval operation;

[0129] S10203. In the retrieval data set, the remaining image-text pairs in the data set are used as a training set for objective function training.

[0130] The acquisition of depth image features in step S103 of this embodiment is specifically as follows:

[0131] S10301, preprocessing image modality data: uniformly cropping and reshaping the input image to a size of H×H pixels, then converting the reshaped image into a flattened two-dimensional plane image block, and normalizing the flattened two-dimensional plane image block to eliminate differences between images;

[0132] S10302. Extract deep image features: Load the blip2-vision-model model onto the GPU through the pre-trained model BLIP-2, and use the blip2-vision-model model to extract image deep features, thereby capturing high-level semantic information of the image.

[0133] The details of obtaining the deep text features in step S104 of this embodiment are as follows:

[0134] S10401, preprocessing text modal data: performing preprocessing operations of text cleaning, text segmentation and text vectorization on the input text;

[0135] S10402. Extract deep text features: Load the blip2-vision-model model onto the GPU through the pre-trained model BLIP-2, and use the blip2-vision-model model to extract text deep features, thereby capturing high-level semantic information of the text.

[0136] As attached Figure 2 As shown, the construction of a multimodal hash retrieval model based on a multi-view center structure in step S2 of this embodiment is specifically as follows:

[0137] S201, modality-specific prototype learning: using image modality deep multilayer perceptron and text modality deep multilayer perceptron to extract refined features from the corresponding modalities respectively, and calculating the average value of the image modality and text modality refined features to obtain modality-specific prototypes, and then obtaining the unique features of the image modality and the unique features of the text modality, and then realizing prototype alignment and prototype separation through the prototype contrast loss function;

[0138] S202, multimodal integrated prototype learning: transform the heterogeneous modal feature set into a unified relaxed hash coding space, merge the refined features of the image modality and the text modality using a concatenation operation, apply a linear transformation layer to project in the feature space and adjust the scale of the features, map the adjusted refined joint representation into a continuous hash code containing rich multimodal semantic information through the Tanh activation function, and then minimize the distance between the hash code and the hash code matching prototype while maximizing the distance to other prototypes through the integrated prototype cross entropy loss function;

[0139] S203, multi-view semantic enhanced hash learning: define a similarity matrix, and use the pairwise semantic similarity information to optimize the generation of hash codes through the similarity preserving loss function, introduce a structural similarity matrix, and use the structural similarity information to optimize the generation of hash codes through the structured neighborhood preserving loss function to capture the structural information in multimodal data.

[0140] As attached Figure 5 As shown, the modality-specific prototype learning in step S201 of this embodiment is specifically as follows:

[0141] S20101. Refine the image modality features: The high-level semantic information of the image is passed through a deep multi-layer perceptron specific to the image modality to obtain the image refinement features and map them into a common 512-dimensional space. The formula is as follows:

[0142]

[0143] in, represents the refined feature matrix extracted from the image modality; x represents the xth sample; X represents the input image modality data; θ xrepresents the xth trainable parameter; represents the real number domain; k×m means that the matrix has k rows and m columns. After being processed by the MLP function, the features will be transformed into a space with the same dimension m;

[0144] S20102. Refine text modality features: The high-level semantic information of the text is passed through a deep multi-layer perceptron specific to the text modality to obtain image refinement features and map them into a common 512-dimensional space. The formula is as follows:

[0145]

[0146] in, represents the refined feature matrix extracted from the image modality; y represents the yth sample; Y represents the input image modality data; θ y represents the yth trainable parameter; Represents the real number domain; k×m means that the matrix has k rows and m columns. After being processed by the MLP function, the features will be converted into a space with the same dimension m; Deep Multilayer Perceptron MLP x and MLP y They all contain an input layer, a series of carefully designed hidden layers, and an output layer. In order to enhance the generalization ability of the model and alleviate the overfitting problem, a powerful regularization technique, Dropout, is integrated into the model architecture.

[0147] S20103. Obtaining an image modality specific prototype: Calculate the average value of the refined features of the text modality samples through a specific prototype generator to obtain a modality specific prototype. The formula is as follows:

[0148]

[0149] Among them, N c Indicates the number of samples belonging to the cth category; function The value is 1 when the i sample belongs to the cth category, otherwise it is 0;

[0150] S20104. Obtain text modality specific prototype: Calculate the average value of the refined features of the image modality samples through the specific prototype generator to obtain the modality specific prototype. The formula is as follows:

[0151]

[0152] Among them, N c Indicates the number of samples belonging to the cth category; function The value is 1 when the i sample belongs to the cth category, otherwise it is 0;

[0153] S20105. Introduce prototype contrast loss function to perform prototype learning, and realize prototype alignment and prototype separation;

[0154] Among them, the prototype contrast loss function is conceptually decomposed into two key components: prototype alignment and prototype separation, and the formula is as follows:

[0155]

[0156] The prototype alignment part makes the prototypes of the same category closer across modalities: the model is optimized by minimizing the distance between the input prototype and the output feature, so that the optimized model can better learn the feature representation of each category. The dot product between the feature vector of category C and the model prediction vector, after adjustment by the temperature parameter, reflects the model's confidence in the current category;

[0157] The prototype separation part separates prototypes of different categories: the model is optimized by maximizing the distance between different categories, so that the optimized model can better distinguish different categories. It is a logarithmic summation operation that involves summing the exponential scores of all non-C categories and then taking the logarithm, which encourages the optimization model to distinguish the feature vector of the current category from the feature vector of non-current categories.

[0158] As attached Figure 6 As shown, the multimodal integrated prototype learning in step S202 of this embodiment is specifically as follows:

[0159] S20201. Refined joint representation: The refined features of the image modality and the text modality are combined using a concatenation operation, and a linear transformation layer is applied to project and rescale the features in the feature space. The formula is as follows:

[0160]

[0161] Among them, Z represents the fused feature matrix; and represents the refined feature matrix extracted from each modality; ⊕ represents the connection operation; W represents the weight matrix; b represents the bias vector;

[0162] S20202, Mapping Hash Code: Apply the Tanh activation function to the refined joint representation to map the refined joint representation into a continuous hash code containing rich multimodal semantic information. The formula is as follows:

[0163]

[0164] Among them, Z represents the representation after refinement and union, Tanh represents the nonlinear hyperbolic tangent function;

[0165] S20203. Define multimodal integrated prototype learning. The formula is as follows:

[0166] P c ={p1,p1,…,p C}∈R k×C ;

[0167] Among them, p i represents the corresponding prototype; k represents the total number of prototypes;

[0168] S20204. Introduce integrated prototype cross entropy loss function: Introduce the integrated prototype cross entropy loss function to optimize the model parameters. The integrated prototype cross entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance with other prototypes.

[0169] As attached Figure 7 As shown, the multi-view semantic enhanced hash learning in step S203 of this embodiment is specifically as follows:

[0170] S20301. Define a similarity matrix: Use the pairwise semantic similarity information as a guide when generating hash codes and introduce a similarity preservation loss function L s , the similarity matrix is ​​integrated into the learning of hash codes to ensure that the generated hash codes can reflect the semantic similarity between samples. The similarity matrix function formula is as follows:

[0171]

[0172] Among them, S ij Represents the similarity matrix, which is used to quantify the semantic similarity between sample i and sample j, l i and l j is the eigenvector of sample i and sample j; e represents the base of natural logarithm, and T represents transpose;

[0173] S20302. Define a structural similarity matrix: introduce the structured neighborhood preservation loss function L q The structural information in multimodal data is captured, and the structural similarity matrix is ​​introduced into the learning of hash codes. By minimizing the structured neighborhood preservation loss function, the hash codes of similar samples are made close to each other, while avoiding excessive similarity between dissimilar samples. The structural similarity matrix function formula is as follows:

[0174]

[0175] Among them, T ij represents the structural similarity matrix, which is used to quantify the similarity of the consecutive hash codes between sample i and sample j; h i and h j represents the consecutive hash codes of sample i and sample j; ||hi ||2||h j ||2 represents the hash code h i and h j The Euclidean norm of ;

[0176] The structural similarity matrix is ​​calculated as the normalized dot product of consecutive hash codes, and its values ​​are adjusted between 0 and 1 to ensure that the values ​​are in a meaningful range.

[0177] As attached Figure 3 As shown, the training model in step S3 of this embodiment is specifically as follows:

[0178] S301, constructing loss function: The goal of the multimodal hash retrieval model based on the multi-view center structure is to construct a loss function to train the multimodal hash retrieval model based on the multi-view center structure by integrating the prototype contrast loss function, the integrated prototype cross entropy loss function, the similarity preservation loss function and the structured neighborhood preservation loss function into a composite function, as follows:

[0179] S30101. Minimize the prototype contrast loss function L c , the formula is as follows:

[0180]

[0181] Among them, C represents the total number of categories of all samples; the logarithmic part is the softmax function, which is used to convert the input linear score into a probability distribution and Represent the input feature and output feature respectively; τ is a temperature parameter that controls the scale of the distribution; the temperature parameter controls the smoothness of the softmax function, thereby affecting the generalization ability of the model. The smaller the value of the temperature parameter, the sharper the distribution of the output of the softmax function, that is, the higher the confidence of the multimodal hash retrieval model based on the multi-view center structure for any category; conversely, the larger the value of the temperature parameter, the smoother the distribution of the output of the softmax function, and the more uncertain the prediction of the multimodal hash retrieval model based on the multi-view center structure;

[0182] S30102. Minimize the prototype cross entropy loss function, the formula is as follows:

[0183]

[0184] Where D represents the measured Euclidean distance; N represents the number of samples; p i Represents the corresponding prototype; the logarithm is a softmax operation, which is used to calculate the probability that the i-th sample belongs to its matching prototype. In this way, the multimodal hash retrieval model based on the multi-view center structure learns the relative distance between each sample and its matching prototype;

[0185] S30103, minimize the similarity preservation loss function L s , the formula is as follows:

[0186]

[0187] Among them, ||·|| represents the L2 norm; cos(h i ,h j ) represents the continuous hash codes h of sample i and sample j i and h j The cosine similarity between ij represents the similarity matrix;

[0188] S30104, minimize the structured neighborhood preservation loss function L q , the formula is as follows:

[0189]

[0190] Where M represents the number of samples involved in the retrieval process; [Δ] ij represents a binary indicator, which is equal to 1 if samples i and j belong to the same category and 0 otherwise; the structural similarity matrix is ​​calculated as the normalized dot product of consecutive hash codes, adjusted to ensure that the values ​​are in a meaningful range, i.e., between 0 and 1;

[0191] S30105, integrate the prototype contrast loss function, the integrated prototype cross entropy loss function, the similarity preservation loss function and the structured neighborhood preservation loss function into a composite loss function, minimize the composite loss function, so as to obtain high-quality hash codes and achieve more efficient multimodal hash retrieval. The specific function is as follows:

[0192] L=αL c +βL s +λL p +ηL q ;

[0193] Among them, α, β, λ, and η are hyperparameters used to balance the contribution of each loss term to achieve the best balance;

[0194] S302. Optimize model: Use the Adam algorithm as the optimization function of the multimodal hash retrieval model based on the multi-view center structure; wherein the learning rate parameter is set to 0.0001, and other hyperparameters use the default values ​​in PyTorch.

[0195] In this embodiment, MAP value is selected as the evaluation index.

[0196] The model of this embodiment is experimented on three widely used public datasets: MirFlickr-25K dataset, MS COCO dataset and NUS-SIDE dataset. The experiment shows that the model of this embodiment is better than the existing model. The comparison of the experimental results MAP is shown in the following table:

[0197]

[0198] The model of this embodiment is compared with the existing model, and it can be seen that the performance of this embodiment is the best compared with other methods.

[0199] Embodiment 2:

[0200] This embodiment provides a multi-modal hash retrieval system based on a multi-view center structure, which is used to implement the multi-modal hash retrieval method based on a multi-view center structure in Embodiment 1; the system includes:

[0201] The multimodal dataset construction module is used to obtain image modality and text modality datasets, process the image modality and text modality datasets, and then perform deep feature extraction on the image modality and text modality datasets through the pre-trained model BLIP-2 to obtain deep image features and deep text features; at the same time, the training set, test set and retrieval set are constructed according to the proportion, and then combined to generate the final dataset;

[0202] A multimodal hash retrieval model construction module based on a multi-view center structure is used to construct a multimodal hash retrieval model based on a multi-view center structure through a neural network and deep learning technology; wherein the multimodal hash retrieval model based on a multi-view center structure refers to a multimodal hash model with a prototype as the center structure;

[0203] A model training module is used to train a multimodal hash retrieval model based on a multi-view center structure using the final data set through a composite loss function;

[0204] Among them, the multimodal hash retrieval model construction module based on the multi-view center structure includes:

[0205] The modality-specific prototype learning submodule is used to extract refined features from the corresponding modalities using the image modality deep multilayer perceptron and the text modality deep multilayer perceptron respectively, and calculate the average value of the image modality and text modality refined features to obtain the modality-specific prototype, and then obtain the unique features of the image modality and the unique features of the text modality, and then realize prototype alignment and prototype separation through the prototype contrast loss function;

[0206] The multimodal integrated prototype learning submodule is used to transform the heterogeneous modal feature set into a unified relaxed hash coding space, merge the refined features of the image modality and the text modality using a connection operation, apply a linear transformation layer to project in the feature space and adjust the scale of the features, and map the adjusted refined joint representation into a continuous hash code containing rich multimodal semantic information through the Tanh activation function. The integrated prototype cross entropy loss function is then used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance to other prototypes.

[0207] The multi-view semantically enhanced hash learning submodule is used to define a similarity matrix and utilize the pairwise semantic similarity information to optimize the generation of hash codes through a similarity-preserving loss function. A structural similarity matrix is ​​introduced and the structural similarity information is utilized to optimize the generation of hash codes through a structured neighborhood-preserving loss function to capture the structural information in multimodal data.

[0208] Embodiment 3:

[0209] This embodiment also provides an electronic device, including: a memory and a processor;

[0210] Wherein, the memory stores computer-executable instructions;

[0211] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the multimodal hash retrieval method based on the multi-view center structure in any embodiment of the present invention.

[0212] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0213] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0214] Embodiment 4:

[0215] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor to enable the processor to execute the multimodal hash retrieval method based on a multi-view center structure in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0216] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0217] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0218] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0219] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal hash retrieval method based on a multi-view center structure, characterized in that: The method is as follows: Constructing a multimodal dataset: Obtain image modality and text modality datasets, process them, and then perform deep feature extraction on them using the pre-trained model BLIP-2 to obtain deep image features and deep text features. At the same time, construct training sets, test sets, and retrieval sets in proportion, and then combine them to generate the final dataset. Construct a multimodal hash retrieval model based on a multi-view center structure: Construct a multimodal hash retrieval model based on a multi-view center structure through neural network and deep learning technology; Training model: Use the final dataset to train the multimodal hash retrieval model based on the multi-view center structure through a composite loss function; Among them, the construction of a multimodal hash retrieval model based on a multi-view center structure is as follows: Modality-specific prototype learning: Use image modality deep multilayer perceptron and text modality deep multilayer perceptron to extract refined features from the corresponding modalities respectively, and calculate the average value of the image modality and text modality refined features to obtain the modality-specific prototype, and then obtain the unique features of the image modality and the unique features of the text modality, and then realize prototype alignment and prototype separation through the prototype contrast loss function; Multimodal ensemble prototype learning: The heterogeneous modal feature set is converted into a unified relaxed hash coding space, and the refined features of the image modality and the text modality are merged using a connection operation. A linear transformation layer is applied to project and adjust the scale of the features in the feature space. The refined joint representation after adjustment is mapped into a continuous hash code containing rich multimodal semantic information through the Tanh activation function, and then the integrated prototype cross entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance to other prototypes. Multi-view semantically enhanced hash learning: define a similarity matrix and use the pairwise semantic similarity information to optimize the generation of hash codes through a similarity-preserving loss function. Introduce a structural similarity matrix and use the structural similarity information to optimize the generation of hash codes through a structured neighborhood-preserving loss function to capture the structural information in multimodal data.

2. The multimodal hash retrieval method based on a multi-view center structure according to claim 1 is characterized in that: The construction of a multimodal dataset is as follows: Obtain public datasets: Obtain and download public datasets containing image modalities and text modalities from public network resources as original datasets; public datasets include MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset; Divide the downloaded original data set; Use the pre-trained model BLIP-2 as the image encoder to extract deep features from the image modality and obtain deep image features; Use the pre-trained model BLIP-2 as the text encoder to perform deep feature extraction on the text modality and obtain deep text features; The data division of the downloaded original data set is as follows: Randomly select any part of image-text pairs on the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset as the test set to evaluate the test performance; Excluding the test set image-text pairs in the MirFlickr-25K dataset, MS COCO, and NUS-SIDE dataset, a portion of the remaining image-text pairs is randomly selected as the retrieval dataset for the retrieval operation; In the retrieval dataset, the remaining image-text pairs in the dataset are used as training sets for objective function training.

3. The multimodal hash retrieval method based on a multi-view center structure according to claim 1 is characterized in that: The specific steps for obtaining deep image features are as follows: Preprocessing image modality data: uniformly crop and reshape the input image to H×H pixel size, then convert the reshaped image into a flat two-dimensional plane image block, and normalize the flat two-dimensional plane image block to eliminate the difference between images; Extract deep image features: Load the blip2-vision-model model onto the GPU through the pre-trained model BLIP-2, and use the blip2-vision-model model to extract image deep features, thereby capturing high-level semantic information of the image; The details of obtaining deep text features are as follows: Preprocessing text modal data: preprocessing the input text by text cleaning, text segmentation, and text vectorization; Extract deep text features: Load the blip2-vision-model model onto the GPU through the pre-trained model BLIP-2, and use the blip2-vision-model model to extract text deep features to capture high-level semantic information of the text.

4. The multimodal hash retrieval method based on a multi-view center structure according to claim 1 is characterized in that: Modality-specific prototype learning is as follows: Refine the image modality features: The high-level semantic information of the image is passed through a deep multi-layer perceptron specific to the image modality to obtain the image refinement features and map them into a common 512-dimensional space. The formula is as follows: in, represents the refined feature matrix extracted from the image modality; x represents the xth sample; X represents the input image modality data; θ x represents the xth trainable parameter; represents the real number domain; k×m means that the matrix has k rows and m columns. After being processed by the MLP function, the features will be transformed into a space with the same dimension m; Refined text modality features: The high-level semantic information of the text is passed through a deep multi-layer perceptron specific to the text modality to obtain image refinement features and map them into a common 512-dimensional space. The formula is as follows: in, represents the refined feature matrix extracted from the image modality; y represents the yth sample; Y represents the input image modality data; θ y represents the yth trainable parameter; Represents the real number domain; k×m means that the matrix has k rows and m columns. After being processed by the MLP function, the features will be converted into a space with the same dimension m; Deep Multilayer Perceptron MLP x and MLP y They all contain an input layer, a series of hidden layers, and an output layer; Obtaining image modality-specific prototypes: The prototype generator calculates the average value of the refined features of the text modality samples to obtain the modality-specific prototype. The formula is as follows: Among them, N c Indicates the number of samples belonging to the cth category; function The value is 1 when the i sample belongs to the cth category, otherwise it is 0; Obtaining text modality-specific prototypes: The prototype generator calculates the average value of the refined features of the image modality samples to obtain the modality-specific prototype. The formula is as follows: Among them, N c Indicates the number of samples belonging to the cth category; function The value is 1 when the i sample belongs to the cth category, otherwise it is 0; The prototype contrast loss function is introduced to perform prototype learning and realize prototype alignment and prototype separation; Among them, the prototype contrast loss function is decomposed into two key components: prototype alignment and prototype separation, and the formula is as follows: The prototype alignment part makes the prototypes of the same category closer across modalities: the model is optimized by minimizing the distance between the input prototype and the output feature, so that the optimized model can learn the feature representation of each category. The dot product between the feature vector of category C and the model prediction vector, after adjustment by the temperature parameter, reflects the model's confidence in the current category; The prototype separation part separates prototypes of different categories: the model is optimized by maximizing the distance between different categories, so that the optimized model can better distinguish different categories. It is a logarithmic summation operation that involves summing the exponential scores of all non-C categories and then taking the logarithm, which encourages the optimization model to distinguish the feature vector of the current category from the feature vector of non-current categories.

5. The multimodal hash retrieval method based on a multi-view center structure according to claim 1 is characterized in that: The specifics of multimodal integration prototype learning are as follows: Refined joint representation: The refined features of the image modality and the text modality are combined using a concatenation operation, and a linear transformation layer is applied to project and rescale the features in the feature space. The formula is as follows: Among them, Z represents the fused feature matrix; and represents the refined feature matrix extracted from the respective modality; represents the connection operation; W represents the weight matrix; b represents the bias vector; Mapping hash code: Apply the Tanh activation function to the refined joint representation to map the refined joint representation into a continuous hash code containing rich multimodal semantic information. The formula is as follows: Among them, Z represents the representation after refinement and union, Tanh represents the nonlinear hyperbolic tangent function; Define multimodal integrated prototype learning, the formula is as follows: P c ={p1,p1,...,p C }∈R k×C ; Among them, p i represents the corresponding prototype; k represents the total number of prototypes; Introducing the integrated prototype cross entropy loss function: The integrated prototype cross entropy loss function is introduced to optimize the model parameters. The integrated prototype cross entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance with other prototypes.

6. The multimodal hash retrieval method based on a multi-view center structure according to claim 1 is characterized in that: The multi-view semantic enhanced hash learning is as follows: Define a similarity matrix: Use pairwise semantic similarity information as a guide when generating hash codes and introduce a similarity preservation loss function L s , the similarity matrix is ​​integrated into the learning of hash codes to ensure that the generated hash codes can reflect the semantic similarity between samples. The similarity matrix function formula is as follows: Among them, S ij Represents the similarity matrix, which is used to quantify the semantic similarity between sample i and sample j, l i and l j is the eigenvector of sample i and sample j; e represents the base of natural logarithm, and T represents transpose; Define a structural similarity matrix: Introduce the structured neighborhood preservation loss function L q The structural information in multimodal data is captured, and the structural similarity matrix is ​​introduced into the learning of hash codes. By minimizing the structured neighborhood preservation loss function, the hash codes of similar samples are made close to each other, while avoiding excessive similarity between dissimilar samples. The structural similarity matrix function formula is as follows: Among them, T ij represents the structural similarity matrix, which is used to quantify the similarity of the continuous hash codes between sample i and sample j; h i and h j represents the consecutive hash codes of sample i and sample j; ||h i ||2||h j ||2 represents the hash code h i and h j The Euclidean norm of ; The structural similarity matrix is ​​calculated as the normalized dot product of consecutive hash codes, and its values ​​are adjusted between 0 and 1 to ensure that the values ​​are in a meaningful range.

7. The multimodal hash retrieval method based on a multi-view center structure according to any one of claims 1 to 6, characterized in that: The training model is as follows: Constructing loss function: The goal of the multimodal hash retrieval model based on the multi-view center structure is to construct a loss function to train the multimodal hash retrieval model based on the multi-view center structure by integrating the prototype contrast loss function, the integrated prototype cross entropy loss function, the similarity preservation loss function and the structured neighborhood preservation loss function into a composite function, as follows: Minimize the prototype contrast loss function L c , the formula is as follows: Among them, C represents the total number of categories of all samples; the logarithmic part is the softmax function, which is used to convert the input linear score into a probability distribution and Represent the input feature and output feature respectively; T is a temperature parameter that controls the scale of the distribution; the temperature parameter controls the smoothness of the softmax function, thereby affecting the generalization ability of the model. The smaller the value of the temperature parameter, the sharper the distribution of the output of the softmax function, that is, the higher the confidence of the multimodal hash retrieval model based on the multi-view center structure for any category; conversely, the larger the value of the temperature parameter, the smoother the distribution of the output of the softmax function, and the more uncertain the prediction of the multimodal hash retrieval model based on the multi-view center structure; Minimize the prototype cross entropy loss function, the formula is as follows: Where D represents the measured Euclidean distance; N represents the number of samples; p i Represents the corresponding prototype; the logarithm is a softmax operation, which is used to calculate the probability that the i-th sample belongs to its matching prototype. In this way, the multimodal hash retrieval model based on the multi-view center structure learns the relative distance between each sample and its matching prototype; Minimize the similarity preserving loss function L s , the formula is as follows: Among them, ||·|| represents the L2 norm; cos(h i ,h j ) represents the continuous hash codes h of sample i and sample j i and h j The cosine similarity between ij represents the similarity matrix; Minimize the structured neighborhood preserving loss function L q , the formula is as follows: Where M represents the number of samples involved in the retrieval process; [Δ] ij represents a binary indicator, which is equal to 1 if samples i and j belong to the same category and 0 otherwise; the structural similarity matrix is ​​calculated as the normalized dot product of consecutive hash codes, adjusted to ensure that the values ​​are in a meaningful range, i.e., between 0 and 1; The prototype contrast loss function, the integrated prototype cross entropy loss function, the similarity preservation loss function and the structured neighborhood preservation loss function are integrated into a composite loss function, which is minimized to obtain high-quality hash codes and achieve more efficient multimodal hash retrieval. The specific function is as follows: L=αL c +βL s +λL p +ηL q ; Among them, α, β, λ, and η are hyperparameters used to balance the contribution of each loss term to achieve the best balance; Optimization model: The Adam algorithm is used as the optimization function of the multimodal hash retrieval model based on the multi-view center structure; the learning rate parameter is set to 0.0001, and other hyperparameters use the default values ​​in PyTorch.

8. A multimodal hash retrieval system based on a multi-view center structure, characterized in that: The system is used to implement the multimodal hash retrieval method based on a multi-view center structure as described in any one of claims 1 to 7; the system includes: The multimodal dataset construction module is used to obtain image modality and text modality datasets, process the image modality and text modality datasets, and then perform deep feature extraction on the image modality and text modality datasets through the pre-trained model BLIP-2 to obtain deep image features and deep text features; at the same time, the training set, test set and retrieval set are constructed according to the proportion, and then combined to generate the final dataset; A multimodal hash retrieval model construction module based on a multi-view center structure is used to construct a multimodal hash retrieval model based on a multi-view center structure through neural networks and deep learning technology; A model training module is used to train a multimodal hash retrieval model based on a multi-view center structure using the final data set through a composite loss function; Among them, the multimodal hash retrieval model construction module based on the multi-view center structure includes: The modality-specific prototype learning submodule is used to extract refined features from the corresponding modalities using the image modality deep multilayer perceptron and the text modality deep multilayer perceptron respectively, and calculate the average value of the image modality and text modality refined features to obtain the modality-specific prototype, and then obtain the unique features of the image modality and the unique features of the text modality, and then realize prototype alignment and prototype separation through the prototype contrast loss function; The multimodal integrated prototype learning submodule is used to transform the heterogeneous modal feature set into a unified relaxed hash coding space, merge the refined features of the image modality and the text modality using a connection operation, apply a linear transformation layer to project in the feature space and adjust the scale of the features, and map the adjusted refined joint representation into a continuous hash code containing rich multimodal semantic information through the Tanh activation function. The integrated prototype cross entropy loss function is then used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance to other prototypes. The multi-view semantically enhanced hash learning submodule is used to define a similarity matrix and utilize the pairwise semantic similarity information to optimize the generation of hash codes through a similarity-preserving loss function. A structural similarity matrix is ​​introduced and the structural similarity information is utilized to optimize the generation of hash codes through a structured neighborhood-preserving loss function to capture the structural information in multimodal data.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the multimodal hash retrieval method based on the multi-view center structure as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the multimodal hash retrieval method based on a multi-view center structure as described in any one of claims 1 to 7.

Images