Multimodal hash retrieval method, system, device and medium based on multi-view center structure

Abstract

This invention discloses a multimodal hash retrieval method, system, device, and medium based on a multi-viewpoint central structure, belonging to the fields of artificial intelligence and multimodal hash retrieval technology. The technical problem this invention aims to solve is achieving a balance between intra-class compactness and inter-class separability during multimodal hash retrieval. The technical solution adopted is as follows: constructing a multimodal dataset; constructing a multimodal hash retrieval model based on a multi-viewpoint central structure; and training the model. Specifically, constructing the multimodal hash retrieval model based on a multi-viewpoint central structure involves: modality-specific prototype learning: using an image modality deep multilayer perceptron and a text modality deep multilayer perceptron to extract refined features from the corresponding modalities, and calculating the average value of the refined features of the image modality and the text modality to obtain modality-specific prototypes, thereby obtaining unique features of the image modality and unique features of the text modality; multimodal ensemble class prototype learning; and multi-view semantic enhancement hash learning.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and multimodal hash retrieval technology, specifically a multimodal hash retrieval method, system, device, and medium based on a multi-view central structure. Background Technology

[0002] With the continuous advancement of technology and the arrival of the information age, multimodal data, encompassing images, text, video, and audio, is widely utilized, resulting in an explosive growth in related data. However, the progress of data retrieval methods has lagged behind, and traditional modal data retrieval methods can no longer meet the increasingly diverse needs of multimodal data retrieval. In recent years, the increasing volume of multimodal data and the urgent need for efficient retrieval have driven increased interest in multimodal retrieval research. Multimodal data is characterized by its relevance and complementarity; that is, data from different modalities have different characteristics, structures, and representations, but they may describe the same entity or event from the perspective of their respective modalities, while data within a modality also exhibits one or more relationships with each other. Therefore, multimodal data retrieval needs to be able to find relevant information across different modalities while ensuring its effectiveness within each modality, achieving accurate and efficient retrieval.

[0003] Unsupervised multimodal hashing is an information retrieval technique that aims to map data from different modalities into a common binary space. It transforms high-dimensional multimodal data into compact and information-rich binary codes, enabling efficient retrieval while preserving the inherent semantic structure of the data. This achieves cost-effective and computationally efficient multimodal retrieval. Unsupervised multimodal hashing maintains similarity between different modalities without requiring additional labels or semantic information, thus reducing costs and effort and providing greater flexibility in practical applications.

[0004] In recent years, many multimodal hash retrieval models have been applied. Among them, the unsupervised cross-modal hash retrieval method that integrates multiple instance relationships proposes a novel framework. This method constructs a similarity matrix based on instances within each training batch and processes these relationship graphs through graph reasoning methods to enhance the semantic association between instances, demonstrating good performance. Deep Graph Neighborhood Consistency Preservation Network is another unsupervised cross-modal hashing method. It considers the graph nearest neighbor relationship between data and its neighbors, preserves the real-valued information of the data through semi-real-valued and semi-binary optimization strategies, and converts it into binary hash codes. It is suitable for cross-modal retrieval on large-scale datasets. The deep unsupervised hashing method based on latent semantic components proposes a novel deep semantic component hashing framework. This method treats semantic components as latent variables under the expectation-maximization algorithm framework and designs an iterative algorithm with the maximum likelihood of the training data as the objective, which helps the hash model achieve excellent performance. Despite significant progress in multimodal hashing methods, the multimodal fusion methods mentioned above still suffer from problems such as coarse-grained feature limitations, oversimplified fusion strategies, and neglect of structural information. Existing work still faces major challenges in ensuring intra-class compactness and inter-class separability, and the models generally exhibit obvious inter-class overlap and intra-class dispersion problems. Summary of the Invention

[0005] The technical objective of this invention is to provide a multimodal hash retrieval method, system, device, and medium based on a multi-view central structure to address the problem of achieving a balance between intra-class compactness and inter-class separability during multimodal hash retrieval.

[0006] The technical objective of this invention is achieved as follows: a multimodal hash retrieval method based on a multi-view central structure, the specific method of which is as follows:

[0007] Constructing a multimodal dataset: Obtain image modality and text modality datasets, process them, and then use the pre-trained model BLIP-2 to extract deep image features and deep text features from the image modality and text modality datasets; at the same time, construct training set, test set and retrieval set according to proportions, and then combine them to generate the final dataset;

[0008] Constructing a multimodal hash retrieval model based on a multi-view central structure: A multimodal hash retrieval model based on a multi-view central structure is constructed using neural networks and deep learning techniques; where, the multimodal hash retrieval model based on a multi-view central structure refers to a multimodal hash model with a central structure as its prototype;

[0009] Training the model: The multimodal hash retrieval model based on the multi-view center structure is trained using the final dataset through a composite loss function;

[0010] The specific details of constructing a multimodal hash retrieval model based on a multi-view center structure are as follows:

[0011] Modality-specific prototype learning: Image modality deep multilayer perceptron and text modality deep multilayer perceptron are used to extract refined features from the corresponding modalities, and the average value of the refined features of the image modality and the text modality is calculated to obtain modality-specific prototypes. In this way, unique features of the image modality and unique features of the text modality are obtained. Prototype alignment and prototype separation are then achieved through the prototype contrast loss function.

[0012] Multimodal ensemble prototype learning: It transforms heterogeneous modal feature sets into a unified relaxed hash coding space, uses connection operations to merge the refined features of image modality and text modality, applies a linear transformation layer to project in the feature space and adjust the scale of the features, maps the refined joint representation to a continuous hash code containing rich multimodal semantic information through the Tanh activation function, and then uses the ensemble prototype cross-entropy loss function to minimize the distance between hash codes and hash code matching prototypes while maximizing the distance with other prototypes.

[0013] Multi-view semantic augmented hash learning: Define a similarity matrix and use pairwise semantic similarity information to optimize hash code generation through a similarity-preserving loss function. Introduce a structural similarity matrix and use structural similarity information to optimize hash code generation through a structured neighborhood-preserving loss function to capture structural information in multimodal data.

[0014] As a preferred option, the multimodal dataset is constructed as follows:

[0015] Obtain public datasets: Obtain and download public datasets containing image and text modalities from public online resources as the original datasets; among them, public datasets include the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset;

[0016] The downloaded raw dataset is partitioned.

[0017] The pre-trained model BLIP-2 is used as an image encoder to extract deep features from image modalities and obtain deep image features.

[0018] The pre-trained model BLIP-2 is used as a text encoder to extract deep features from the text modality and obtain deep text features.

[0019] The specific data partitioning of the downloaded original dataset is as follows:

[0020] Randomly select a subset of image-text pairs from the MirFlickr-25K dataset, MS COCO, and NUS-SIDE dataset as the test set to evaluate test performance;

[0021] Excluding the test set image-text pairs from the MirFlickr-25K, MS COCO, and NUS-SIDE datasets, a portion of the remaining image-text pairs is randomly selected as the retrieval dataset for retrieval operations.

[0022] In the retrieved dataset, the remaining image-text pairs in the dataset are used as the training set for training the objective function.

[0023] As a preferred method, the depth image features are obtained as follows:

[0024] Preprocessing image modal data: The input image is uniformly cropped, reshaped and adjusted to H×H pixel size, then the reshaped image is converted into a flat two-dimensional planar image block, and the flat two-dimensional planar image block is normalized to eliminate the differences between images;

[0025] Extracting deep image features: The blip2-vision-model is loaded onto the GPU using the pre-trained model BLIP-2, and the blip2-vision-model is used to extract deep image features, thereby capturing the high-level semantic information of the image;

[0026] The specific steps for obtaining deep text features are as follows:

[0027] Preprocessing text modal data: Preprocessing the input text by performing text cleaning, text segmentation, and text vectorization.

[0028] Extracting deep text features: The blip2-vision-model is loaded onto the GPU using the pre-trained BLIP-2 model. The blip2-vision-model is then used to extract deep text features, thereby capturing the high-level semantic information of the text.

[0029] As a preferred approach, modality-specific prototype learning is as follows:

[0030] Refining image modal features: The high-level semantic information of the image is passed through a deep multilayer perceptron specific to the image modality to obtain image refinement features, which are then mapped to a common 512-dimensional space, as shown in the following formula:

[0031]

[0032] in, Represents the refined feature matrix extracted from the image modality; x represents the x-th sample; X represents the input image modality data; θx This represents the x-th trainable parameter; Represents the real number field; k×m indicates that the matrix has k rows and m columns. After processing by the MLP function, the features will be transformed into a space with the same dimension m.

[0033] Refining text modal features: The high-level semantic information of the text is processed through a text modality-specific deep multilayer perceptron to obtain refined text features, which are then mapped to a common 512-dimensional space, as shown in the following formula:

[0034]

[0035] in, θ represents the refined feature matrix extracted from the text modality; y represents the y-th sample; Y represents the input text modality data; θ y This represents the y-th trainable parameter; Represents the real number field; k×m indicates that the matrix has k rows and m columns. After processing by the MLP function, the features will be transformed into a space with the same dimension m; Deep Multilayer Perceptron (MLP) x and MLP y Each model contains an input layer, a series of carefully designed hidden layers, and an output layer. To enhance the model's generalization ability and mitigate overfitting, a powerful regularization technique, Dropout, is incorporated into the model's architecture.

[0036] Obtaining a modality-specific prototype: The modality-specific prototype is obtained by calculating the average value of the refined features of the image modality samples using a specific prototype generator, as shown in the following formula:

[0037]

[0038] Where, N c This represents the number of samples belonging to the c-th category; the function... The value is 1 if sample i belongs to the c-th category, and 0 otherwise.

[0039] Obtaining a text modality-specific prototype: The modality-specific prototype is obtained by calculating the average of the refined features of the text modality samples using a specific prototype generator, as shown in the following formula:

[0040]

[0041] Where, N c This represents the number of samples belonging to the c-th category; the function... The value is 1 if sample i belongs to the c-th category, and 0 otherwise.

[0042] A prototype contrast loss function is introduced to learn prototypes, achieving prototype alignment and prototype separation.

[0043] The prototype contrast loss function is conceptually decomposed into two key components: prototype alignment and prototype separation, as shown in the following formula:

[0044]

[0045] The prototype alignment component brings prototypes of the same category closer together across modalities: by minimizing the distance between the input prototype and the output features, the model is optimized, enabling it to better learn the feature representations for each category. The dot product between the feature vector of category c and the model prediction vector, after being adjusted for the temperature parameter, reflects the model's confidence in the current category.

[0046] The prototype separation section separates prototypes of different categories: it optimizes the model by maximizing the distance between different categories, enabling the optimized model to better distinguish between different categories. It is a logarithmic summation operation that involves summing the exponential scores of all non-C categories and then taking the logarithm, which encourages the optimization model to distinguish the feature vectors of the current category from the feature vectors of non-C categories.

[0047] As a preferred approach, multimodal ensemble class prototype learning is as follows:

[0048] Refined Joint Representation: This method combines the refined features of the image and text modalities using a join operation. A linear transformation layer is then applied to project the features into the feature space and adjust their scale. The formula is as follows:

[0049]

[0050] Where Z represents the fused feature matrix; and This represents the refined feature matrix extracted from each modality; This represents the join operation; W represents the weight matrix; b represents the bias vector.

[0051] Mapping Hash Code: Applying the Tanh activation function to the refined joint representation maps it to a continuous hash code containing rich multimodal semantic information, as shown in the following formula:

[0052]

[0053] Where Z represents the refined joint representation, and Tanh represents the nonlinear hyperbolic tangent function;

[0054] The prototype learning of multimodal inheritance classes is defined by the following formula:

[0055] P C ={p1,p2,…,p c}∈Rk×C ;

[0056] Where, p c This represents the corresponding prototype; k represents the total number of prototypes.

[0057] Introducing an ensemble prototype cross-entropy loss function: The ensemble prototype cross-entropy loss function is introduced to optimize model parameters. By using the ensemble prototype cross-entropy loss function, the distance between hash codes and hash code matching prototypes is minimized while the distance with other prototypes is maximized.

[0058] As a preferred approach, multi-view semantic augmented hash learning is specifically as follows:

[0059] Define a similarity matrix: introduce a similarity-preserving loss function L, using pairwise semantic similarity information as guidance when generating hash codes. s The similarity matrix is ​​integrated into the learning of hash codes to ensure that the generated hash codes can reflect the semantic similarity between samples. The formula for the similarity matrix function is as follows:

[0060]

[0061] Among them, S ij The similarity matrix represents the semantic similarity between sample i and sample j. i and l j is the feature vector of sample i and sample j; e represents the base of the natural logarithm, and T represents the transpose;

[0062] Define a structural similarity matrix: Introduce a structured neighborhood preservation loss function L q Structural information is captured from multimodal data, and the structural similarity matrix is ​​introduced into hash code learning. By minimizing the structured neighborhood preservation loss function, the hash codes of similar samples are made close to each other, while avoiding excessive similarity between dissimilar samples. The formula for the structural similarity matrix function is as follows:

[0063]

[0064] Among them, T ij The structural similarity matrix is ​​used to quantify the similarity of consecutive hash codes between sample i and sample j; h i and h j Represents the consecutive hash codes of sample i and sample j; ||h i ||2||h j ||2 represents the hash code h i and h j The Euclidean norm;

[0065] The structural similarity matrix is ​​calculated as a normalized dot product of consecutive hash codes, with its value adjusted between 0 and 1 to ensure that the value is within a meaningful range.

[0066] More preferably, the training model is specifically as follows:

[0067] Constructing the Loss Function: The goal of the multimodal hash retrieval model based on the multi-view center structure is to construct a loss function by integrating the prototype contrast loss function, the prototype cross-entropy loss function, the similarity preservation loss function, and the structured neighborhood preservation loss function into a composite function. The model is trained using this loss function, as detailed below:

[0068] Minimize the prototype contrast loss function L c The formula is as follows:

[0069]

[0070] Where C represents the total number of categories in all samples; the logarithmic part is the softmax function, used to transform the linear scores of the input into a probability distribution. and τ represents the image modality-specific prototype and the text modality-specific prototype, respectively; τ is a temperature parameter that controls the distribution scale; the temperature parameter controls the smoothness of the softmax function, thus affecting the generalization ability of the model. The smaller the value of the temperature parameter, the sharper the distribution of the softmax function output, that is, the higher the confidence of the multi-view center structure-based multimodal hash retrieval model for any category; conversely, the larger the value of the temperature parameter, the smoother the distribution of the softmax function output, and the more uncertain the prediction of the multi-view center structure-based multimodal hash retrieval model.

[0071] The formula for minimizing the prototype cross-entropy loss function is as follows:

[0072]

[0073] Where D represents the measured Euclidean distance; N represents the sample size; p i The corresponding prototype is represented; the logarithm is a softmax operation used to calculate the probability that the i-th sample belongs to its matching prototype. In this way, the multimodal hash retrieval model based on the multi-view center structure learns the relative distance between each sample and its matching prototype.

[0074] Minimize the similarity preservation loss function L s The formula is as follows:

[0075]

[0076] Where ||·|| represents the L2 norm; cos(hi ,h j ) represents the consecutive hash codes h of samples i and j. i and h j Cosine similarity between them; S ij Represents the similarity matrix;

[0077] Minimize the structured neighborhood preservation loss function L q The formula is as follows:

[0078]

[0079] Where M represents the number of samples involved in the retrieval process; [Δ] ij This represents a binary indicator, which is equal to 1 if samples i and j belong to the same category, and equal to 0 otherwise; the structural similarity matrix is ​​calculated as a normalized dot product of consecutive hash codes, adjusted to ensure that the values ​​are within a meaningful range, i.e., between 0 and 1;

[0080] The prototype contrast loss function, the ensemble prototype cross-entropy loss function, the similarity preservation loss function, and the structured neighborhood preservation loss function are integrated into a composite loss function. Minimizing this composite loss function yields high-quality hash codes, enabling more efficient multimodal hash retrieval. The specific function is as follows:

[0081] L=αL c +βL s +λL p +ηL q ;

[0082] Among them, α, β, λ and η are hyperparameters used to balance the contributions of each loss term to achieve the optimal balance;

[0083] Model optimization: The Adam algorithm is used as the optimization function for the multimodal hash retrieval model based on the multi-view center structure; the learning rate parameter is set to 0.0001, and the default values ​​in PyTorch are used for other hyperparameters.

[0084] A multimodal hash retrieval system based on a multi-view central structure is provided. This system is used to implement the multimodal hash retrieval method based on a multi-view central structure as described above. The system includes:

[0085] The multimodal dataset construction module is used to acquire image modality and text modality datasets, process the image modality and text modality datasets, and then use the pre-trained model BLIP-2 to extract deep image features and deep text features from the image modality and text modality datasets; at the same time, training set, test set and retrieval set are constructed according to the proportion, and then combined to generate the final dataset;

[0086] The module for building a multimodal hash retrieval model based on a multi-view central structure is used to construct a multimodal hash retrieval model based on a multi-view central structure using neural networks and deep learning techniques; where the multimodal hash retrieval model based on a multi-view central structure refers to a multimodal hash model whose prototype is a central structure.

[0087] The model training module is used to train the multimodal hash retrieval model based on the multi-view center structure using the final dataset and a composite loss function.

[0088] The multimodal hash retrieval model construction module based on the multi-view center structure includes:

[0089] The modality-specific prototype learning submodule is used to extract refined features from the corresponding modalities using image modality deep multilayer perceptron and text modality deep multilayer perceptron, respectively, and calculate the average value of the refined features of image modality and text modality to obtain modality-specific prototypes, thereby obtaining unique features of image modality and unique features of text modality, and then achieving prototype alignment and prototype separation through prototype contrast loss function;

[0090] The multimodal ensemble prototype learning submodule is used to transform heterogeneous modal feature sets into a unified relaxed hash coding space. It uses connection operations to merge the refined features of image modality and text modality, applies a linear transformation layer to project in the feature space and adjust the scale of the features, and maps the refined joint representation to a continuous hash code containing rich multimodal semantic information through the Tanh activation function. Finally, the ensemble prototype cross-entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance with other prototypes.

[0091] The multi-view semantic augmented hash learning submodule is used to define a similarity matrix and optimize the generation of hash codes through a similarity-preserving loss function using pairwise semantic similarity information. It also introduces a structural similarity matrix and optimizes the generation of hash codes through a structured neighborhood-preserving loss function using structural similarity information, thereby capturing structural information in multimodal data.

[0092] An electronic device includes: a memory and at least one processor;

[0093] The memory contains computer programs;

[0094] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multimodal hash retrieval method based on the multi-view center structure as described above.

[0095] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the multimodal hash retrieval method based on a multi-view central structure as described above.

[0096] The multimodal hash retrieval method, system, device, and medium based on a multi-view central structure of the present invention have the following advantages:

[0097] (i) This invention solves the problem that existing work is difficult to balance between intra-class compactness and inter-class separability. It learns the prototype of a specific modality by using intra-modal class information to ensure the effective fusion of unique features of each modality, and learns the prototype of a multimodal integrated class across modalities, thereby significantly enhancing inter-class separability and solving the problems of inter-class overlap and intra-class dispersion in multimodal hashing, thus ensuring the balance between intra-class compactness and inter-class separability.

[0098] (ii) This invention addresses the limitations of traditional methods by recognizing the importance of subtle local differences within a single modality and global information spanning multiple modalities, achieving a significant improvement in retrieval performance compared to the most advanced existing methods. Its core idea is to seamlessly integrate the advantages of modal prototypes and integrated prototypes, and to comprehensively explore semantic information from multiple perspectives.

[0099] (III) This invention proposes a novel multimodal hashing framework. First, a set of consecutive hash codes is learned. These hash codes are optimized in a continuous space to maintain the semantic similarity between multimodal samples. Then, a sign function is applied to this set of consecutive hash codes to obtain a strict binary hash code. This ensures that the generated binary hash code not only effectively compresses high-dimensional multimodal samples, but also maintains the discriminative ability to achieve accurate retrieval.

[0100] (iv) This invention uses modality-specific deep multilayer perceptrons to identify that each modality in heterogeneous data carries its unique modality-specific information, extracts refined features from each modality, captures the subtle attributes and features inherent in different data modalities more meticulously, and ensures that the unique information contained in each modality is fully utilized. By using these modality-specific deep multilayer perceptrons, features from different modalities are mapped to a common unified feature space. This transformation simplifies the process of heterogeneous modality alignment and paves the way for the complex and effective fusion of heterogeneous modalities.

[0101] (V) This invention proposes a method to seamlessly integrate local and global class information by leveraging the advantages of specific modal prototypes and integrated prototypes, thereby promoting the combination of intramodal and intermodal correlations. It mainly consists of two parts: prototype alignment and prototype separation. The former aims to bring prototypes of the same category across modalities closer together (prototype alignment), while the latter pushes prototypes of different categories apart (prototype separation). By effectively utilizing these prototypes, the semantic depth and distinguishability of hash codes are significantly enhanced.

[0102] (vi) This invention proposes a prototype contrast loss function, which effectively aligns cross-modal prototypes within a class by optimizing the prototype contrast loss, while separating prototypes of different classes.

[0103] (vii) This invention proposes an integrated prototype cross-entropy loss function, which minimizes the distance between the hash code and its matching prototype, while maximizing the distance with other prototypes, thereby promoting intra-class cohesion and enhancing inter-class separation.

[0104] (viii) This invention proposes a similarity preservation loss function, which integrates the similarity matrix into the learning of hash codes and enhances the discriminative ability of hash codes by utilizing pairwise semantic similarity information;

[0105] (ix) This invention proposes a structured neighborhood preservation loss function, which aims to minimize the ambiguity between adjacent samples, ensure that the generated hash codes accurately maintain the similarity structure, promote the hash codes of similar samples to be close to each other, and avoid excessive similarity between dissimilar samples.

[0106] (x) This invention proposes to optimize the hash code generation process by minimizing a composite loss function, thereby solving the limitations caused by coarse-grained feature constraints, simplified fusion strategies and ignoring structural information, and enhancing the retrieval performance of multimodal hashing. Attached Figure Description

[0107] The invention will be further described below with reference to the accompanying drawings.

[0108] Appendix Figure 1 The flowchart shows the multimodal hash retrieval method based on a multi-view center structure.

[0109] Appendix Figure 2 A flowchart illustrating the process of constructing a multimodal hash retrieval model based on a multi-view center structure;

[0110] Appendix Figure 3 A flowchart for training the model;

[0111] Appendix Figure 4 A flowchart for constructing a multimodal dataset;

[0112] Appendix Figure 5Flowchart for modality-specific prototype learning;

[0113] Appendix Figure 6 A flowchart for learning prototypes of multimodal integration classes;

[0114] Appendix Figure 7 A flowchart for learning multi-view semantic augmented hashing. Detailed Implementation

[0115] The multimodal hash retrieval method, system, device, and medium based on a multi-view central structure of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0116] Example 1:

[0117] As attached Figure 1 As shown in the figure, this embodiment provides a multimodal hash retrieval method based on a multi-view center structure, which is as follows:

[0118] S1. Construct a multimodal dataset: Obtain image modality and text modality datasets, process the image modality and text modality datasets, and then use the pre-trained model BLIP-2 to extract deep image features and deep text features from the image modality and text modality datasets; at the same time, construct training set, test set and retrieval set according to proportion, and then combine them to generate the final dataset;

[0119] S2. Construct a multimodal hash retrieval model based on a multi-view central structure: Construct a multimodal hash retrieval model based on a multi-view central structure using neural networks and deep learning techniques; where, the multimodal hash retrieval model based on a multi-view central structure refers to a multimodal hash model with a prototype central structure;

[0120] S3. Training the model: Using the final dataset, train the multimodal hash retrieval model based on the multi-view center structure using a composite loss function.

[0121] As attached Figure 4 As shown, the construction of the multimodal dataset in step S1 of this embodiment is as follows:

[0122] S101. Obtain public datasets: Obtain and download public datasets containing image and text modalities from public network resources as the original datasets; among them, public datasets include the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset;

[0123] S102. Divide the downloaded raw dataset into data segments;

[0124] S103. Use the pre-trained model BLIP-2 as an image encoder to extract deep features from the image modalities and obtain deep image features;

[0125] S104. Use the pre-trained model BLIP-2 as a text encoder to extract deep features from the text modality and obtain deep text features.

[0126] In this embodiment, step S102 involves dividing the downloaded raw dataset into data segments as follows:

[0127] S10201. Randomly select a portion of image-text pairs from the MirFlickr-25K dataset, MS COCO, and NUS-SIDE dataset as the test set to evaluate test performance;

[0128] S10202. Exclude the test set image-text pairs from the MirFlickr-25K dataset, MS COCO, and NUS-SIDE dataset, and randomly select a portion of the remaining image-text pairs as the retrieval dataset for retrieval operations.

[0129] S10203. In the retrieved dataset, the remaining image-text pairs in the dataset are used as the training set for training the objective function.

[0130] The specific steps for obtaining depth image features in step S103 of this embodiment are as follows:

[0131] S10301, Preprocessing image modal data: uniformly crop and reshape the input image to H×H pixel size, then convert the reshaped image into a flat two-dimensional planar image block, and normalize the flat two-dimensional planar image block to eliminate the differences between images.

[0132] S10302. Extracting deep image features: The blip2-vision-model is loaded onto the GPU using the pre-trained model BLIP-2. The blip2-vision-model is then used to extract deep image features, thereby capturing the high-level semantic information of the image.

[0133] The specific steps for obtaining deep text features in step S104 of this embodiment are as follows:

[0134] S10401, Preprocessing text modal data: Preprocessing the input text by text cleaning, text segmentation and text vectorization;

[0135] S10402. Extracting deep text features: The blip2-vision-model is loaded onto the GPU using the pre-trained model BLIP-2. The blip2-vision-model is then used to extract deep text features, thereby capturing the high-level semantic information of the text.

[0136] As attached Figure 2 As shown, the construction of the multimodal hash retrieval model based on the multi-view center structure in step S2 of this embodiment is as follows:

[0137] S201, Modality-Specific Prototype Learning: Using image modality deep multilayer perceptron and text modality deep multilayer perceptron, respectively, refinement features are extracted from the corresponding modalities. The average value of the refinement features of the image modality and the text modality is calculated to obtain modality-specific prototypes. In turn, unique features of the image modality and unique features of the text modality are obtained. Prototype alignment and prototype separation are then achieved through the prototype contrast loss function.

[0138] S202, Multimodal Ensemble Prototype Learning: Transforms heterogeneous modal feature sets into a unified relaxed hash coding space, merges the refined features of image and text modalities using connection operations, applies a linear transformation layer to project and adjust the scale of features in the feature space, maps the adjusted refined joint representation to a continuous hash code containing rich multimodal semantic information through the Tanh activation function, and then minimizes the distance between hash codes and hash code matching prototypes while maximizing the distance with other prototypes through the ensemble prototype cross-entropy loss function.

[0139] S203, Multi-view Semantic Enhanced Hash Learning: Define a similarity matrix and use pairwise semantic similarity information to optimize hash code generation through a similarity-preserving loss function. Introduce a structural similarity matrix and use structural similarity information to optimize hash code generation through a structured neighborhood-preserving loss function to capture structural information in multimodal data.

[0140] As attached Figure 5 As shown, the modality-specific prototype learning in step S201 of this embodiment is as follows:

[0141] S20101. Refining Image Modal Features: The high-level semantic information of the image is processed through a deep multilayer perceptron specific to the image modality to obtain image refinement features, which are then mapped to a common 512-dimensional space, as shown in the following formula:

[0142]

[0143] in, Represents the refined feature matrix extracted from the image modality; x represents the x-th sample; X represents the input image modality data; θ xThis represents the x-th trainable parameter; Represents the real number field; k×m indicates that the matrix has k rows and m columns. After processing by the MLP function, the features will be transformed into a space with the same dimension m.

[0144] S20102. Refining Text Modal Features: The high-level semantic information of the text is processed through a text modality-specific deep multilayer perceptron to obtain refined text features, which are then mapped to a common 512-dimensional space, as shown in the following formula:

[0145]

[0146] in, θ represents the refined feature matrix extracted from the text modality; y represents the y-th sample; Y represents the input text modality data; θ y This represents the y-th trainable parameter; Represents the real number field; k×m indicates that the matrix has k rows and m columns. After processing by the MLP function, the features will be transformed into a space with the same dimension m; Deep Multilayer Perceptron (MLP) x and MLP y Each model contains an input layer, a series of carefully designed hidden layers, and an output layer. To enhance the model's generalization ability and mitigate overfitting, a powerful regularization technique, Dropout, is incorporated into the model's architecture.

[0147] S20103. Obtaining the modality-specific prototype: The modality-specific prototype is obtained by calculating the average value of the refined features of the image modality samples using a specific prototype generator, as shown in the following formula:

[0148]

[0149] Where, N c This represents the number of samples belonging to the c-th category; the function... The value is 1 if sample i belongs to the c-th category, and 0 otherwise.

[0150] S20104. Obtaining the text modality-specific prototype: Calculate the average value of the refined features of the text modality samples using a specific prototype generator to obtain the modality-specific prototype, as shown in the following formula:

[0151]

[0152] Where, N c This represents the number of samples belonging to the c-th category; the function... The value is 1 if sample i belongs to the c-th category, and 0 otherwise.

[0153] S20105. Introduce the prototype contrast loss function to perform prototype learning, and realize prototype alignment and prototype separation.

[0154] The prototype contrast loss function is conceptually decomposed into two key components: prototype alignment and prototype separation, as shown in the following formula:

[0155]

[0156] The prototype alignment component brings prototypes of the same category closer together across modalities: by minimizing the distance between the input prototype and the output features, the model is optimized, enabling it to better learn the feature representations for each category. The dot product between the image modality-specific prototype and the text modality-specific prototype of category c, after being adjusted for the temperature parameter, reflects the model's confidence in the current category.

[0157] The prototype separation section separates prototypes of different categories: it optimizes the model by maximizing the distance between different categories, enabling the optimized model to better distinguish between different categories. It is a logarithmic summation operation that involves summing the exponential scores of all non-C categories and then taking the logarithm, which encourages the optimization model to distinguish the feature vectors of the current category from the feature vectors of non-C categories.

[0158] As attached Figure 6 As shown, the multimodal integration class prototype learning in step S202 of this embodiment is as follows:

[0159] S20201, Thinning Joint Representation: Thinning features from the image and text modalities are merged using a join operation. A linear transformation layer is applied to project the features into the feature space and adjust their scale. The formula is as follows:

[0160]

[0161] Where Z represents the fused feature matrix; and ⊕ represents the refined feature matrix extracted from each modality; ⊕ represents the join operation; W represents the weight matrix; b represents the bias vector;

[0162] S20202, Mapped Hash Code: Applying the Tanh activation function to the refined joint representation maps it to a continuous hash code containing rich multimodal semantic information, as shown in the following formula:

[0163]

[0164] Where Z represents the refined joint representation, and Tanh represents the nonlinear hyperbolic tangent function;

[0165] S20203. Define multimodal ensemble class prototype learning, the formula is as follows:

[0166] P C ={p1,p2,…,p c}∈R k×C ;

[0167] Where, p c This represents the corresponding prototype; k represents the total number of prototypes.

[0168] S20204. Introducing the ensemble prototype cross-entropy loss function: The ensemble prototype cross-entropy loss function is introduced to optimize the model parameters. By using the ensemble prototype cross-entropy loss function, the distance between hash codes and hash code matching prototypes is minimized while the distance with other prototypes is maximized.

[0169] As attached Figure 7 As shown, the multi-view semantic enhancement hash learning in step S203 of this embodiment is as follows:

[0170] S20301. Define a similarity matrix: using pairwise semantic similarity information as guidance when generating hash codes, introduce a similarity preservation loss function L. s The similarity matrix is ​​integrated into the learning of hash codes to ensure that the generated hash codes can reflect the semantic similarity between samples. The formula for the similarity matrix function is as follows:

[0171]

[0172] Among them, S ij The similarity matrix represents the semantic similarity between sample i and sample j. i and l j is the feature vector of sample i and sample j; e represents the base of the natural logarithm, and T represents the transpose;

[0173] S20302. Define a structural similarity matrix: Introduce a structured neighborhood preservation loss function L. q Structural information is captured from multimodal data, and the structural similarity matrix is ​​introduced into hash code learning. By minimizing the structured neighborhood preservation loss function, the hash codes of similar samples are made close to each other, while avoiding excessive similarity between dissimilar samples. The formula for the structural similarity matrix function is as follows:

[0174]

[0175] Among them, T ij The structural similarity matrix is ​​used to quantify the similarity of consecutive hash codes between sample i and sample j; h i and h j Represents the consecutive hash codes of sample i and sample j; ||hi ||2||h j ||2 represents the hash code h i and h j The Euclidean norm;

[0176] The structural similarity matrix is ​​calculated as a normalized dot product of consecutive hash codes, with its value adjusted between 0 and 1 to ensure that the value is within a meaningful range.

[0177] As attached Figure 3 As shown, the specific training model in step S3 of this embodiment is as follows:

[0178] S301. Constructing the Loss Function: The goal of the multimodal hash retrieval model based on the multi-view center structure is to construct a loss function by integrating the prototype contrast loss function, the prototype cross-entropy loss function, the similarity preservation loss function, and the structured neighborhood preservation loss function into a composite function. The specific details are as follows:

[0179] S30101, Minimize the prototype contrast loss function L c The formula is as follows:

[0180]

[0181] Where C represents the total number of categories in all samples; the logarithmic part is the softmax function, used to transform the linear scores of the input into a probability distribution. and τ represents the image modality-specific prototype and the text modality-specific prototype, respectively; τ is a temperature parameter that controls the distribution scale; the temperature parameter controls the smoothness of the softmax function, thus affecting the generalization ability of the model. The smaller the value of the temperature parameter, the sharper the distribution of the softmax function output, that is, the higher the confidence of the multi-view center structure-based multimodal hash retrieval model for any category; conversely, the larger the value of the temperature parameter, the smoother the distribution of the softmax function output, and the more uncertain the prediction of the multi-view center structure-based multimodal hash retrieval model.

[0182] S30102. Minimize the prototype cross-entropy loss function, as shown in the following formula:

[0183]

[0184] Where D represents the measured Euclidean distance; N represents the sample size; p i The corresponding prototype is represented; the logarithm is a softmax operation used to calculate the probability that the i-th sample belongs to its matching prototype. In this way, the multimodal hash retrieval model based on the multi-view center structure learns the relative distance between each sample and its matching prototype.

[0185] S30103, Minimize the similarity preservation loss function L s The formula is as follows:

[0186]

[0187] Where ||·|| represents the L2 norm; cos(h i ,h j ) represents the consecutive hash codes h of samples i and j. i and h j Cosine similarity between them; S ij Represents the similarity matrix;

[0188] S30104, Minimize the structured neighborhood preservation loss function L q The formula is as follows:

[0189]

[0190] Where M represents the number of samples involved in the retrieval process; [Δ] ij This represents a binary indicator, which is equal to 1 if samples i and j belong to the same category, and equal to 0 otherwise; the structural similarity matrix is ​​calculated as a normalized dot product of consecutive hash codes, adjusted to ensure that the values ​​are within a meaningful range, i.e., between 0 and 1;

[0191] S30105 integrates the prototype contrast loss function, the prototype cross-entropy loss function, the similarity preservation loss function, and the structured neighborhood preservation loss function into a composite loss function. Minimizing this composite loss function yields high-quality hash codes, enabling more efficient multimodal hash retrieval. The specific function is as follows:

[0192] L=αL c +βL s +λL p +ηL q ;

[0193] Among them, α, β, λ and η are hyperparameters used to balance the contributions of each loss term to achieve the optimal balance;

[0194] S302. Optimize the model: Use the Adam algorithm as the optimization function for the multimodal hash retrieval model based on the multi-view center structure; the learning rate parameter is set to 0.0001, and the other hyperparameters can use the default values ​​in PyTorch.

[0195] In this embodiment, the MAP value is selected as the evaluation index.

[0196] The model in this embodiment was tested on three widely used public datasets: MirFlickr-25K, MS COCO, and NUS-SIDE. The experiments show that the model in this embodiment outperforms existing models. The comparison of the MAP results is shown in the table below:

[0197]

[0198] The model in this embodiment was compared with existing models, and it can be seen that the performance of this embodiment is the best compared with other methods.

[0199] Example 2:

[0200] This embodiment provides a multimodal hash retrieval system based on a multi-view central structure. This system is used to implement the multimodal hash retrieval method based on a multi-view central structure in Embodiment 1. The system includes:

[0201] The multimodal dataset construction module is used to acquire image modality and text modality datasets, process the image modality and text modality datasets, and then use the pre-trained model BLIP-2 to extract deep image features and deep text features from the image modality and text modality datasets; at the same time, training set, test set and retrieval set are constructed according to the proportion, and then combined to generate the final dataset;

[0202] The module for building a multimodal hash retrieval model based on a multi-view central structure is used to construct a multimodal hash retrieval model based on a multi-view central structure using neural networks and deep learning techniques; where the multimodal hash retrieval model based on a multi-view central structure refers to a multimodal hash model whose prototype is a central structure.

[0203] The model training module is used to train the multimodal hash retrieval model based on the multi-view center structure using the final dataset and a composite loss function.

[0204] The multimodal hash retrieval model construction module based on the multi-view center structure includes:

[0205] The modality-specific prototype learning submodule is used to extract refined features from the corresponding modalities using image modality deep multilayer perceptron and text modality deep multilayer perceptron, respectively, and calculate the average value of the refined features of image modality and text modality to obtain modality-specific prototypes, thereby obtaining unique features of image modality and unique features of text modality, and then achieving prototype alignment and prototype separation through prototype contrast loss function;

[0206] The multimodal ensemble prototype learning submodule is used to transform heterogeneous modal feature sets into a unified relaxed hash coding space. It uses connection operations to merge the refined features of image modality and text modality, applies a linear transformation layer to project in the feature space and adjust the scale of the features, and maps the refined joint representation to a continuous hash code containing rich multimodal semantic information through the Tanh activation function. Finally, the ensemble prototype cross-entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance with other prototypes.

[0207] The multi-view semantic augmented hash learning submodule is used to define a similarity matrix and optimize the generation of hash codes through a similarity-preserving loss function using pairwise semantic similarity information. It also introduces a structural similarity matrix and optimizes the generation of hash codes through a structured neighborhood-preserving loss function using structural similarity information, thereby capturing structural information in multimodal data.

[0208] Example 3:

[0209] This embodiment also provides an electronic device, including: a memory and a processor;

[0210] The memory stores the instructions executed by the computer.

[0211] The processor executes computer execution instructions stored in the memory, causing the processor to perform the multimodal hash retrieval method based on a multi-view center structure in any embodiment of the present invention.

[0212] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0213] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0214] Example 4:

[0215] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the multimodal hash retrieval method based on a multi-view central structure according to any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0216] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0217] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0218] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0219] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal hash retrieval method based on a multi-view center structure, characterized in that, The method is as follows: Constructing a multimodal dataset: Obtain image modality and text modality datasets, process them, and then use the pre-trained model BLIP-2 to extract deep image features and deep text features from the image modality and text modality datasets; at the same time, construct training set, test set and retrieval set according to proportions, and then combine them to generate the final dataset; Constructing a multimodal hash retrieval model based on a multi-view center structure: Constructing a multimodal hash retrieval model based on a multi-view center structure using neural network and deep learning techniques; Training the model: The multimodal hash retrieval model based on the multi-view center structure is trained using the final dataset through a composite loss function; The specific details of constructing a multimodal hash retrieval model based on a multi-view center structure are as follows: Modality-specific prototype learning: Image modality deep multilayer perceptron and text modality deep multilayer perceptron are used to extract refined features from the corresponding modalities, and the average value of the refined features of the image modality and the text modality is calculated to obtain modality-specific prototypes. In this way, unique features of the image modality and unique features of the text modality are obtained. Prototype alignment and prototype separation are then achieved through the prototype contrast loss function. Multimodal ensemble prototype learning: It transforms heterogeneous modal feature sets into a unified relaxed hash coding space, uses connection operations to merge the refined features of image modality and text modality, applies a linear transformation layer to project in the feature space and adjust the scale of the features, maps the refined joint representation to a continuous hash code containing rich multimodal semantic information through the Tanh activation function, and then uses the ensemble prototype cross-entropy loss function to minimize the distance between hash codes and hash code matching prototypes while maximizing the distance with other prototypes. Multi-view semantic augmented hash learning: Define a similarity matrix and use pairwise semantic similarity information to optimize hash code generation through a similarity-preserving loss function. Introduce a structural similarity matrix and use structural similarity information to optimize hash code generation through a structured neighborhood-preserving loss function to capture structural information in multimodal data. The multi-view semantic augmented hash learning is specifically as follows: Define a similarity matrix: introduce a similarity-preserving loss function L, using pairwise semantic similarity information as guidance when generating hash codes. s The similarity matrix is ​​integrated into the learning of hash codes to ensure that the generated hash codes can reflect the semantic similarity between samples. The formula for the similarity matrix function is as follows: Among them, S ij The similarity matrix represents the semantic similarity between sample i and sample j. i and l j is the feature vector of sample i and sample j; e represents the base of the natural logarithm, and T represents the transpose; Define a structural similarity matrix: Introduce a structured neighborhood preservation loss function L q Structural information is captured from multimodal data, and the structural similarity matrix is ​​introduced into hash code learning. By minimizing the structured neighborhood preservation loss function, the hash codes of similar samples are made close to each other, while avoiding excessive similarity between dissimilar samples. The formula for the structural similarity matrix function is as follows: Among them, T ij The structural similarity matrix is ​​used to quantify the similarity of consecutive hash codes between sample i and sample j; h i and h j Represents the consecutive hash codes of sample i and sample j; ||h i ‖2‖h j ‖2 represents the hash code h i and h j The Euclidean norm; The structural similarity matrix is ​​calculated as a normalized dot product of consecutive hash codes, with its value adjusted between 0 and 1 to ensure that the value is within a meaningful range.

2. The multimodal hash retrieval method based on a multi-view center structure according to claim 1, characterized in that, The construction of the multimodal dataset is as follows: Obtain public datasets: Obtain and download public datasets containing image and text modalities from public online resources as the original datasets; among them, public datasets include the MirFlickr-25K dataset, MS COCO and NUS-SIDE dataset; The downloaded raw dataset is partitioned. The pre-trained model BLIP-2 is used as an image encoder to extract deep features from image modalities and obtain deep image features. The pre-trained model BLIP-2 is used as a text encoder to extract deep features from the text modality and obtain deep text features. The specific data partitioning of the downloaded original dataset is as follows: Randomly select a subset of image-text pairs from the MirFlickr-25K dataset, MS COCO, and NUS-SIDE dataset as the test set to evaluate test performance; Excluding the test set image-text pairs from the MirFlickr-25K, MS COCO, and NUS-SIDE datasets, a portion of the remaining image-text pairs is randomly selected as the retrieval dataset for retrieval operations. In the retrieved dataset, the remaining image-text pairs in the dataset are used as the training set for training the objective function.

3. The multimodal hash retrieval method based on a multi-view center structure according to claim 1, characterized in that, The specific steps for obtaining depth image features are as follows: Preprocessing image modal data: The input image is uniformly cropped, reshaped and adjusted to H×H pixel size, then the reshaped image is converted into a flat two-dimensional planar image block, and the flat two-dimensional planar image block is normalized to eliminate the differences between images; Extracting deep image features: The blip2-vision-model is loaded onto the GPU using the pre-trained model BLIP-2, and the blip2-vision-model is used to extract deep image features, thereby capturing the high-level semantic information of the image; The specific steps for obtaining deep text features are as follows: Preprocessing text modal data: Preprocessing the input text by performing text cleaning, text segmentation, and text vectorization. Extracting deep text features: The blip2-vision-model is loaded onto the GPU using the pre-trained BLIP-2 model. The blip2-vision-model is then used to extract deep text features, thereby capturing the high-level semantic information of the text.

4. The multimodal hash retrieval method based on a multi-view center structure according to claim 1, characterized in that, Modality-specific prototype learning is as follows: Refining image modal features: The high-level semantic information of the image is passed through a deep multilayer perceptron specific to the image modality to obtain image refinement features, which are then mapped to a common 512-dimensional space, as shown in the following formula: in, Represents the refined feature matrix extracted from the image modality; x represents the x-th sample; X represents the input image modality data; θ x This represents the x-th trainable parameter; Represents the real number field; k×m indicates that the matrix has k rows and m columns. After processing by the MLP function, the features will be transformed into a space with the same dimension m. Refining text modal features: The high-level semantic information of the text is processed through a text modality-specific deep multilayer perceptron to obtain refined text features, which are then mapped to a common 512-dimensional space, as shown in the following formula: in, θ represents the refined feature matrix extracted from the text modality; y represents the y-th sample; Y represents the input text modality data; θ y This represents the y-th trainable parameter; Represents the real number field; k×m indicates that the matrix has k rows and m columns. After processing by the MLP function, the features will be transformed into a space with the same dimension m; Deep Multilayer Perceptron (MLP) x and MLP y Each layer contains an input layer, a series of hidden layers, and an output layer. Obtaining a modality-specific prototype: The modality-specific prototype is obtained by calculating the average value of the refined features of the image modality samples using a prototype generator, as shown in the following formula: Where, N c This represents the number of samples belonging to the c-th category; the function... The value is 1 if sample i belongs to the c-th category, and 0 otherwise. Obtaining a text modality-specific prototype: The modality-specific prototype is obtained by calculating the average of the refined features of the text modality samples using a prototype generator, as shown in the following formula: Where, N c This represents the number of samples belonging to the c-th category; the function... The value is 1 if sample i belongs to the c-th category, and 0 otherwise. A prototype contrast loss function is introduced to learn prototypes, achieving prototype alignment and prototype separation. The prototype contrast loss function is decomposed into two key components: prototype alignment and prototype separation, as shown in the following formula: Indicates prototype alignment. Indicates prototype separation; prototype alignment portion To make cross-modal prototypes of the same category closer together: optimize the model by minimizing the distance between the input prototype and the output features, enabling the optimized model to learn feature representations for each category. The dot product between the image modality-specific prototype and the text modality-specific prototype of category c, after being adjusted for the temperature parameter, reflects the model's confidence in the current category. Prototype separation part Separate prototypes of different categories: optimize the model by maximizing the distance between different categories, so that the optimized model can better distinguish between different categories. It is a logarithmic summation operation that involves summing the exponential scores of all non-C categories and then taking the logarithm, which encourages the optimization model to distinguish the feature vectors of the current category from the feature vectors of non-C categories.

5. The multimodal hash retrieval method based on a multi-view center structure according to claim 1, characterized in that, The specific steps for learning multimodal inheritance class prototypes are as follows: Refined Joint Representation: This method combines the refined features of the image and text modalities using a join operation. A linear transformation layer is then applied to project the features into the feature space and adjust their scale. The formula is as follows: Where Z represents the fused feature matrix; and This represents the refined feature matrix extracted from each modality; This represents the join operation; W represents the weight matrix; b represents the bias vector. Mapping Hash Code: Applying the Tanh activation function to the refined joint representation maps it to a continuous hash code containing rich multimodal semantic information, as shown in the following formula: Where Z represents the refined joint representation, and Tanh represents the nonlinear hyperbolic tangent function; The prototype learning of multimodal inheritance classes is defined by the following formula: P C ={p1,p2,…,p c }∈R k×C ; Where, p c This represents the corresponding prototype; k represents the total number of prototypes. Introducing an ensemble prototype cross-entropy loss function: The ensemble prototype cross-entropy loss function is introduced to optimize model parameters. By using the ensemble prototype cross-entropy loss function, the distance between hash codes and hash code matching prototypes is minimized while the distance with other prototypes is maximized.

6. The multimodal hash retrieval method based on a multi-view center structure according to any one of claims 1 to 5, characterized in that, The specific training model is as follows: Constructing the Loss Function: The goal of the multimodal hash retrieval model based on the multi-view center structure is to construct a loss function by integrating the prototype contrast loss function, the prototype cross-entropy loss function, the similarity preservation loss function, and the structured neighborhood preservation loss function into a composite function. The model is trained using this loss function, as detailed below: Minimize the prototype contrast loss function L c The formula is as follows: Where C represents the total number of categories in all samples; the logarithmic part is the softmax function, used to transform the linear scores of the input into a probability distribution. and τ represents the image modality-specific prototype and the text modality-specific prototype, respectively; τ is a temperature parameter that controls the distribution scale; the temperature parameter controls the smoothness of the softmax function, thus affecting the generalization ability of the model. The smaller the value of the temperature parameter, the sharper the distribution of the softmax function output, that is, the higher the confidence of the multi-view center structure-based multimodal hash retrieval model for any category; conversely, the larger the value of the temperature parameter, the smoother the distribution of the softmax function output, and the more uncertain the prediction of the multi-view center structure-based multimodal hash retrieval model. The formula for minimizing the prototype cross-entropy loss function is as follows: Where D represents the measured Euclidean distance; N represents the sample size; p i The corresponding prototype is represented; the logarithm is a softmax operation used to calculate the probability that the i-th sample belongs to its matching prototype. In this way, the multimodal hash retrieval model based on the multi-view center structure learns the relative distance between each sample and its matching prototype. Minimize the similarity preservation loss function L s The formula is as follows: Where ||·|| represents the L2 norm; cos(h i ,h j ) represents the consecutive hash codes h of samples i and j. i and h j Cosine similarity between them; S ij Represents the similarity matrix; Minimize the structured neighborhood preservation loss function L q The formula is as follows: Where M represents the number of samples involved in the retrieval process; [Δ] ij This represents a binary indicator, which is equal to 1 if samples i and j belong to the same category, and equal to 0 otherwise; the structural similarity matrix is ​​calculated as a normalized dot product of consecutive hash codes, adjusted to ensure that the values ​​are within a meaningful range, i.e., between 0 and 1; The prototype contrast loss function, the ensemble prototype cross-entropy loss function, the similarity preservation loss function, and the structured neighborhood preservation loss function are integrated into a composite loss function. Minimizing this composite loss function yields high-quality hash codes, enabling more efficient multimodal hash retrieval. The specific function is as follows: L=αL c +βL s +λL p +ηL q ; Among them, α, β, λ and η are hyperparameters used to balance the contributions of each loss term to achieve the optimal balance; Model optimization: The Adam algorithm is used as the optimization function for the multimodal hash retrieval model based on the multi-view center structure.

7. A multimodal hash retrieval system based on a multi-viewpoint central structure, characterized in that, This system is used to implement the multimodal hash retrieval method based on a multi-view center structure as described in any one of claims 1 to 6; the system includes: The multimodal dataset construction module is used to acquire image modality and text modality datasets, process the image modality and text modality datasets, and then use the pre-trained model BLIP-2 to extract deep image features and deep text features from the image modality and text modality datasets; at the same time, training set, test set and retrieval set are constructed according to the proportion, and then combined to generate the final dataset; A multimodal hash retrieval model building module based on a multi-view center structure is used to build a multimodal hash retrieval model based on a multi-view center structure through neural network and deep learning techniques; The model training module is used to train the multimodal hash retrieval model based on the multi-view center structure using the final dataset and a composite loss function. The multimodal hash retrieval model construction module based on the multi-view center structure includes: The modality-specific prototype learning submodule is used to extract refined features from the corresponding modalities using image modality deep multilayer perceptron and text modality deep multilayer perceptron, respectively, and calculate the average value of the refined features of image modality and text modality to obtain modality-specific prototypes, thereby obtaining unique features of image modality and unique features of text modality, and then achieving prototype alignment and prototype separation through prototype contrast loss function; The multimodal ensemble prototype learning submodule is used to transform heterogeneous modal feature sets into a unified relaxed hash coding space. It uses connection operations to merge the refined features of image modality and text modality, applies a linear transformation layer to project in the feature space and adjust the scale of the features, and maps the refined joint representation to a continuous hash code containing rich multimodal semantic information through the Tanh activation function. Finally, the ensemble prototype cross-entropy loss function is used to minimize the distance between the hash code and the hash code matching prototype while maximizing the distance with other prototypes. The multi-view semantic augmented hash learning submodule is used to define a similarity matrix and optimize the generation of hash codes through a similarity-preserving loss function using pairwise semantic similarity information. It also introduces a structural similarity matrix and optimizes the generation of hash codes through a structured neighborhood-preserving loss function using structural similarity information, thereby capturing structural information in multimodal data.

8. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multimodal hash retrieval method based on a multi-view central structure as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the multimodal hash retrieval method based on a multi-view central structure as described in any one of claims 1 to 6.

Images